robots.txt
A plain text file at the root of your website that tells crawlers, AI bots among them, which pages they may read.
robots.txt is a plain text file at the root of your website, at yourbrand.com/robots.txt, that tells crawlers which pages they may read. Search engines check it before they crawl your site, and so do most AI bots. If your robots.txt blocks an AI platform's bot, that platform may not be able to read your pages, and it's less likely to use them in its answers.
How it works
The file is a list of rules. Each group of rules starts with the name of a bot, its user agent, and then says which paths it may or may not read. A * in place of the name means every bot.
For example, this file lets every bot read the whole site, apart from OpenAI's training bot:
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /A bot follows the group that names it. If no group names it, it follows the * group. A rule only works if the bot chooses to follow it, and the AI companies say their bots do.
Training bots and search bots aren't the same
Most AI companies run more than one bot, and each one has its own name. One reads pages to train the company's models. Another reads pages so its assistant can find them and cite them in answers. A third fetches a page when someone asks the assistant to look at it.
So a rule that blocks training doesn't have to block answers. You can let a search bot in and keep a training bot out, or the other way round. Each company lists its bots and what they do on its own page: OpenAI, Google, Anthropic and Perplexity.
Blocking at your host or CDN counts too. A firewall rule that turns away a bot has the same effect as a robots.txt rule, and it doesn't show up in the file.
In AnswerOrbit
The Audit tab in Website reads your robots.txt every day and shows, for each of the 42 AI bots we know from 25 companies, whether it's Allowed, Partly allowed or Blocked. If a bot that could read your site is suddenly blocked, we email you. Every version of the file we've read is kept in History, so you can see which change blocked it. See Website and AI traffic.