Retired crawler tokens
Retired crawler tokens are bot names an AI company no longer uses or lists. robots.txt rules that name them do nothing.
Retired crawler tokens are the names of AI bots that their companies no longer use or list. A rule in your robots.txt that names one doesn't block or allow anything, because no current bot answers to it.
They matter because many sites copied their AI rules from a blog post or another site's file. If those rules name bots that have since been renamed or split up, the site owner thinks they've made a choice that isn't actually in force.
How they come about
AI companies change their bots as their products change. A company with one crawler might split it into three: one for training, one for search, and one that fetches pages when a user asks. The old name drops out of its documentation, and new names appear.
Some names were never official at all. They turned up in a list somewhere and got copied from file to file.
For example, plenty of robots.txt files still carry rules for anthropic-ai and Claude-Web. Anthropic's page about its bots lists ClaudeBot, Claude-User and Claude-SearchBot instead. A site that only blocks the first two names hasn't blocked any of Anthropic's current bots.
How to check yours
Check each AI rule in your file against the company's own page about its bots. Other people's files are how the old names spread. OpenAI, Google and Perplexity each keep one. Then:
- Remove rules for names the company no longer lists
- Add rules for its current bots, if you want them treated differently from the rest
- Decide for each bot whether it's training, search or fetching for a user, and set your rule to match
In AnswerOrbit
The Audit tab in Website and AI traffic reads your robots.txt every day and shows, for each of the 42 AI bots we know from 25 companies, whether it's Allowed, Partly allowed or Blocked, and whether its own rule or your file's general rule decides it. History keeps every version of the file we've read.