Crawl-delay
Crawl-delay is a robots.txt rule asking a bot to wait a set number of seconds between requests to your site.
Crawl-delay is a robots.txt rule that asks a bot to wait a set number of seconds between requests to your site. It's a way to slow a bot down without shutting it out.
User-agent: ClaudeBot
Crawl-delay: 1It's useful when a bot's visits put a strain on your server. Blocking the bot would stop the strain, and also keep your pages out of whatever the bot is for.
Not every bot follows it
Crawl-delay was never part of the core robots.txt rules, and support varies. Google says plainly that it doesn't support crawl-delay. Anthropic says ClaudeBot does, and gives Crawl-delay: 1 as an example. For other bots, check the company's own page about its crawler.
So don't count on Crawl-delay alone to protect your server. If a bot ignores it, your host or CDN can usually limit how fast it may make requests.
Setting the number
The number is seconds between requests, and it adds up fast. A delay of 10 seconds lets a bot read at most 8,640 pages a day. That's plenty for a small site and far too few for a store with 50,000 product pages, which would take nearly six days to read once.
For example, Harrow & Pine's shop slowed down every time an AI bot visited. A delay of 2 seconds for that one bot kept the shop fast and still let the bot read every page within a day.
Set the delay per bot, in that bot's own group, so you don't slow down the bots you're happy with.
In AnswerOrbit
Bot visits in Website shows which AI bots visited your site, which pages they read and which requests failed, once you connect your server logs. That's where you'll see whether one bot is visiting far more than the rest.