LLM crawler
An LLM crawler is a bot an AI company sends to read web pages, to train its language models or to find pages for its answers.
An LLM crawler is a bot that an AI company sends out to read web pages for its language models. Some collect pages to train the models. Others read pages so the company's assistant can find them and cite them in answers. A third kind fetches a page when someone asks the assistant to look at it.
If these bots can't read your site, the platforms they work for know less about you and have fewer of your pages to cite.
Which bots are which
Each AI company names its bots, and most now run more than one. OpenAI, for example, lists GPTBot for training, OAI-SearchBot for search and ChatGPT-User for pages people ask about, on its bots page. Anthropic, Perplexity, Google, Meta and others publish similar lists.
The names matter, because robots.txt rules work by name. Blocking one bot doesn't block the others from the same company.
Should you let them in?
That depends on the bot. Blocking a training bot keeps your pages out of future models, which some publishers want. Blocking a search bot can keep you out of that platform's answers now, which few brands want.
For example, Kestrel Roasters blocks every bot with "AI" in its description to protect its recipes. A month later it notices Perplexity has stopped citing its brewing guides, because the block caught Perplexity's search bot too. It changes the rule to block only the training bots.
Remember your hosting provider or firewall too. Some block AI bots by default, and that block doesn't appear in your robots.txt.
In AnswerOrbit
The Audit tab in Website reads your robots.txt every day and shows, for each of the 42 AI bots we know from 25 companies, its purpose and whether it's Allowed, Partly allowed or Blocked. Connect your server logs and Bot visits shows which ones actually visit. See Website and AI traffic.