Cloudflare AI Crawl Control
Cloudflare AI Crawl Control is Cloudflare's tool for seeing which AI crawlers visit your site and allowing or blocking each one.
Cloudflare AI Crawl Control is a Cloudflare tool that shows which AI crawlers visit your site and lets you allow or block each one. It used to be called AI Audit. If your site runs through Cloudflare, it's one of the places where AI bots can be let in or kept out.
It matters because a block here works whatever your robots.txt says. That's useful when a bot ignores your file. It's a problem when a search bot you wanted gets blocked without anyone noticing.
What it does
According to Cloudflare's documentation, it shows which AI services access your content, lets you allow or block individual crawlers, and tracks whether crawlers follow your robots.txt, with rules to stop the ones that don't. Cloudflare is also testing pay per crawl, which lets you charge a crawler for access. It works on every Cloudflare plan.
robots.txt asks, Crawl Control enforces
A robots.txt rule is a request, and it only works if the bot chooses to follow it. A block at Cloudflare stops the request before it reaches your site.
The catch is that your robots.txt no longer tells the whole story. For example, Fernhill's robots.txt allows every AI search bot. But someone on the team blocked a group of AI crawlers in Cloudflare months ago, and one of them was a search bot. The file looks fine, and the bot still can't get in.
So when you check which bots can read your site, look in both places.
In AnswerOrbit
You can connect Cloudflare as a source of server logs, in Settings under Connections. Bot visits then shows which AI bots visited your site, which pages they read and which requests failed. Cloudflare's free plan caps a Worker's requests each day. The Audit tab reads your robots.txt, which doesn't include blocks made in Cloudflare. See Website and AI traffic.