Bot directory
A bot directory is a public list of known bots, with who runs each one and what it does. Useful for spotting AI bots in your logs.
A bot directory is a public list of known bots: their names, the companies that run them, and what each one does. When you find an unfamiliar name in your server logs or want to write a robots.txt rule, a directory is a quick place to look it up.
It saves time, because the AI companies keep adding bots and splitting old ones into several. A good directory gathers them in one place.
Check the source too
Directories are put together by third parties, and they can be wrong. Some list bots that were renamed or retired. Some list names nobody ever used, copied from another list that guessed. Some describe a bot's purpose loosely, which matters when you're deciding whether to block it.
So use a directory to find a name, then check it on the page the company itself publishes. OpenAI, Google, Anthropic and Perplexity each list their own bots and what they're for.
For example, say your logs show a bot you don't recognize reading your price pages every hour. A directory tells you it belongs to an AI company and says it's a search bot. The company's own page confirms it, and adds that it's the bot that decides whether you can appear in their assistant's answers. That's a bot you'd want to let in.
Names in a rule must match exactly
A robots.txt rule only works when it uses the bot's real name. A rule for a name that doesn't exist does nothing, and you won't get an error telling you so. That's another reason to take names from the company's page.
In AnswerOrbit
The Audit tab in Website works as a directory checked against your own site. It lists the 42 AI bots we know from 25 companies, with each one's purpose, and whether your robots.txt allows it. See Website and AI traffic.