Crawler purpose category
A crawler's purpose category says what a bot reads your pages for: training AI models, search, or fetching a page for a person.
A crawler's purpose category says what a bot reads your pages for. For AI bots, there are three main ones: training bots collect pages to train models, search bots read pages so a platform can find and cite them, and user bots fetch a page because someone asked about it.
Sorting bots by purpose, before sorting them by company, is what lets you make sensible choices. Most decisions about AI bots are really decisions about purpose.
Why purpose matters more than the company
Most AI companies run a bot for each purpose, with a separate name for each. Anthropic, for example, has ClaudeBot for training, Claude-SearchBot for search and Claude-User for fetching pages for people.
Blocking by company is too blunt. If you block every Anthropic bot, you leave training, and you also leave Claude's answers. Blocking by purpose lets you say, for instance, "no training, yes to search" across every company at once.
For example, Harrow & Pine decides it's happy to be found and cited, and would rather its product descriptions weren't used for training. It allows every search bot and user bot, and blocks the training bots by name. Its pages still show up in answers, and its training opt-out is in place.
Where categories get fuzzy
Not every bot fits neatly. Some companies use one bot for several jobs, and then a robots.txt rule can't separate them. For those, newer conventions like Content Signals let you say how your content may be used after it's read.
In AnswerOrbit
The Audit tab in Website lists each AI bot we know with its purpose. Training bots collect pages to train AI models. Search bots read pages so a platform can find them and cite them. Bots marked fetching for a user read a page because someone asked an assistant to look at it. Blocking a training bot keeps your pages out of a model's training, and blocking a search bot can keep them out of that platform's answers. See Website and AI traffic.