Claude-User
Claude-User is the Anthropic bot that fetches a web page when someone's question to Claude needs it.
Claude-User is the bot Anthropic uses to fetch a web page when someone's question to Claude needs it. Someone pastes a link and asks Claude to summarise it, or asks something that needs current information, and Claude-User goes and gets the page. Its visits follow people's questions, one page at a time.
Anthropic's crawler page says that blocking it stops Claude from retrieving your content when a user asks for it, which may lower your visibility in Claude's answers.
How it treats robots.txt
Companies handle their user bots differently. Google says its user-triggered fetchers generally ignore robots.txt, because a person asked for the page. OpenAI says robots.txt rules may not apply to ChatGPT-User for the same reason.
Anthropic says its bots honour robots.txt. So if your file blocks Claude-User, or blocks every bot with a * rule, expect Claude to be kept out of your pages even when someone asks for them by name.
For example, someone asks Claude to compare the office plans of Fernhill and Kestrel, and pastes both pricing pages. If Kestrel's robots.txt blocks Claude-User, Claude can't read Kestrel's page and has to answer from what it already knows, which may be out of date.
In AnswerOrbit
The Audit tab in Website lists each AI bot we know with its purpose. Bots marked as fetching for a user read a page because someone asked an assistant to look at it. If a bot that could read your site is suddenly blocked, we email you. See Website and AI traffic.