Claude-SearchBot
Claude-SearchBot is Anthropic's crawler that reads web pages for the search index behind Claude's answers.
Claude-SearchBot is the Anthropic crawler that reads web pages for the index behind Claude's web search. Anthropic says it uses what it reads to make Claude's search results more relevant and accurate. If it can't read your pages, Claude's search is less likely to find them when someone asks a question you could answer.
Anthropic's crawler page says blocking it stops your content being indexed for search, which may reduce your visibility in Claude's answers.
Search bots and training bots
Most AI companies now keep these jobs apart. OpenAI has OAI-SearchBot for search and GPTBot for training. Anthropic has Claude-SearchBot for search and ClaudeBot for training. That lets you stay out of training and still be found.
For example, Fernhill's robots.txt was set up years ago to block every bot except Googlebot. Nobody noticed that it also kept out Claude-SearchBot. Adding a group for it fixes that:
User-agent: Claude-SearchBot
Allow: /A bot follows the group that names it, so this one rule opens the site to Claude's search while the old * rule still applies to everyone else.
Check your firewall and your host's bot protection too. A block there has the same effect as a robots.txt rule, and it doesn't show up in the file.
In AnswerOrbit
The Audit tab in Website shows, for each AI bot we know, whether your robots.txt allows it and what it's for. If you connect your server logs, Bot visits shows which bots actually came, which pages they read and which requests failed. We check every request before we count it, and requests that fail the check show as Unverified requests. See Website and AI traffic.