ClaudeBot
ClaudeBot is Anthropic's crawler that collects web pages that may be used to train its AI models.
ClaudeBot is Anthropic's training crawler. It collects web pages that may be used to train Anthropic's models, the ones behind Claude. It's one of three Anthropic bots, each with its own job, described on Anthropic's crawler page.
If you block ClaudeBot in your robots.txt, Anthropic says it takes that as a sign to leave your future pages out of its training data.
Blocking it leaves Claude's search alone
Claude's web search uses two other bots, Claude-SearchBot and Claude-User. So blocking ClaudeBot on its own keeps your pages out of training, and Claude can still find your pages and cite them when people ask questions.
For example, this robots.txt keeps ClaudeBot out and lets Anthropic's search bots in:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Whether to block training is your call. Publishers whose writing is what they sell often block it. Many brands are happy to allow it, so that future models know their products and how to describe them.
Slowing it down
If ClaudeBot's visits put a strain on your server, you don't have to block it. Anthropic says ClaudeBot follows the Crawl-delay rule, so you can ask it to wait between requests, for example with Crawl-delay: 1 in its group.
In AnswerOrbit
The Audit tab in Website reads your robots.txt every day and shows, for each AI bot we know, whether it's allowed, partly allowed or blocked, along with its purpose. If a bot that could read your site is suddenly blocked, we email you. See Website and AI traffic.