Crawler
A crawler is a bot that visits web pages and reads them, to build a search index, train an AI model, or fetch a page for someone.
A crawler is a bot that visits websites and reads their pages automatically. Search engines use crawlers to find and index pages. AI companies use them to collect pages for training, to find pages they can cite in answers, and to fetch a page when someone asks about it.
If a crawler can't reach your page, or can't read what's on it, that page can't help you in the answers it feeds.
The three kinds of AI crawler
Most AI companies now run separate bots for separate jobs:
- Training bots collect pages to train AI models. OpenAI's GPTBot is one.
- Search bots read pages so a platform can find them and cite them in answers. OpenAI's OAI-SearchBot is one.
- User bots fetch a page because a person asked about it in a chat. ChatGPT-User is one.
They matter for different reasons. Blocking a training bot keeps your pages out of future models. Blocking a search bot can keep you out of that platform's answers today.
How you control crawlers
Your site's robots.txt file tells crawlers which pages they may read. Each bot is named by its user agent, so you can allow one and block another. Some services, like your hosting provider or firewall, can also block bots on their own, sometimes without you knowing.
For example, a site might block GPTBot to stay out of training, but forget that its firewall also blocks OAI-SearchBot. The site then quietly drops out of ChatGPT's search answers.
Crawlers in AnswerOrbit
The Audit tab on Website reads your robots.txt every day and shows, for each AI bot we know, whether it's allowed, partly allowed or blocked, along with its purpose. If a bot that could read your site is suddenly blocked, we email you. Bot visits shows which bots actually visit, from your server logs.