GPTBot
GPTBot is OpenAI's crawler for collecting pages that may be used to train its AI models. It's separate from the bot that powers ChatGPT search.
GPTBot is the crawler OpenAI uses to collect web pages that may be used to train its AI models. If your robots.txt blocks GPTBot, you're telling OpenAI not to use your content for training.
It's easy to mix up with OpenAI's other bots, and that matters, because GPTBot isn't the one that decides whether ChatGPT search can find your pages.
GPTBot and OpenAI's other bots
OpenAI runs more than one bot, and its bots page explains each:
- GPTBot collects pages for training
- OAI-SearchBot finds pages for ChatGPT's search features
- ChatGPT-User visits a page when a person in ChatGPT asks it to
Each one has its own rule in robots.txt. OpenAI says blocking GPTBot doesn't affect whether your site appears in ChatGPT's search results. That's set by OAI-SearchBot. If you allow both, OpenAI may use one crawl for both purposes.
An example
For example, say Kestrel Roasters doesn't want its blog used for training, but does want ChatGPT to find and cite its pages. Its robots.txt can block one and allow the other:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /If Kestrel had blocked OAI-SearchBot instead, ChatGPT search would stop showing its pages in answers, though OpenAI says they can still appear as navigational links.
In AnswerOrbit
The Audit tab in Website reads your robots.txt every day and shows each AI bot we know as Allowed, Partly allowed or Blocked, with its purpose: training, search, or fetching for a user. If a bot that could read your site is suddenly blocked, we email you. Connect your server logs and the Bot visits tab shows which bots actually came.