Bot
A bot is a program that visits websites on its own. AI companies run bots to train models, find pages to cite, and fetch pages for people.
A bot is a program that visits websites and reads them by itself, with no person clicking. Search engines, AI companies, price checkers, uptime monitors and spammers all run bots. When people in AI search talk about bots, they usually mean the ones AI companies send to read your pages.
Those bots decide what the AI platforms can learn about you. A page their bots can't reach, or can't read, can't help you in their answers.
The kinds of AI bot
Most AI companies run more than one bot, each with its own name and job:
- Training bots collect pages to train the company's models
- Search bots read pages so the assistant can find them and cite them in answers
- Bots that fetch for a user read one page because someone asked the assistant to look at it
Each bot announces itself with a name, its user agent, every time it asks for a page. That's how your robots.txt can give each one its own rules. For example, you might let OpenAI's search bot read your site while keeping its training bot out.
Some bots use a borrowed name
Anyone can write a program and give it the name of a well-known bot. Scrapers do this to get past rules meant for someone else. The big AI companies publish the addresses their bots use, so a request can be checked against that list. OpenAI's is on its bots page.
Bots in AnswerOrbit
The Audit tab in Website shows, for each of the 42 AI bots we know from 25 companies, whether your robots.txt lets it in, and what it's for. Connect your server logs and the Bot visits tab shows which bots actually came. We check every request before we count it, and ones that fail the check are shown as unverified. See Website and AI traffic.