OAI-SearchBot
OAI-SearchBot is OpenAI's search bot. It reads web pages so ChatGPT can find them and show them in its search answers.
OAI-SearchBot is OpenAI's search bot. It reads web pages so that ChatGPT's search features can find them, show them and link to them in answers. It's the OpenAI bot that decides whether your pages can be cited in ChatGPT.
If your site blocks it, OpenAI says your pages won't be shown in ChatGPT search answers. That makes it one of the most important bots to check in your robots.txt.
One of several OpenAI bots
OpenAI runs more than one bot, each with its own name, and lists them on its bots page:
- OAI-SearchBot reads pages for ChatGPT's search.
- GPTBot reads pages that may be used to train OpenAI's models.
- ChatGPT-User visits a page when someone using ChatGPT asks it to.
- OAI-AdsBot checks pages that advertisers submit as ads on ChatGPT.
OpenAI says each setting is independent of the others. So you can keep GPTBot out, to keep your pages out of training, and still let OAI-SearchBot in, so ChatGPT can cite you.
An example
Say Fernhill Coffee's robots.txt was written in a hurry and blocks every OpenAI bot:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /Fernhill wanted to stay out of training, but it has also kept its pages out of ChatGPT's search answers. When someone asks ChatGPT for a good coffee subscription, it can't cite Fernhill's pages. Removing the OAI-SearchBot group fixes it. OpenAI says changes for search can take about 24 hours to take effect.
Check your host and CDN as well. A firewall rule that turns the bot away blocks it just as surely as robots.txt does.
OAI-SearchBot in AnswerOrbit
The Audit tab in Website reads your robots.txt every day and shows whether OAI-SearchBot and the other AI bots we know are allowed, partly allowed or blocked. If a bot that could read your site is suddenly blocked, we email you. Connect your server logs and the Bot visits tab shows when it actually visited, and which pages it read.