Crawl-to-refer ratio
The crawl-to-refer ratio compares how many pages an AI company's bots read on a site with how many visitors it sends back.
The crawl-to-refer ratio compares how many pages a company's bots read on a website with how many visitors it sends back. Cloudflare uses the term in its data on AI bots and referrals, where it measures how many pages a platform crawls compared with how often it drives people to a site.
For example, if an AI company's bots read 3,000 pages on your site in a month, and its assistant sends you 6 visitors, its crawl-to-refer ratio is 500 to 1.
Why people talk about it
Web search ran on a deal: you let search engines read your pages, and they send you visitors. The crawl-to-refer ratio shows how far a company has moved from that deal. A low ratio means the reading is paid back in visits. A high ratio means a lot of reading and few visitors.
Publishers point to the ratio when they argue for being paid for their content, or for blocking bots that give little back.
What drives it
The kind of bot matters most. Training bots read pages to build models, and never send anyone back directly. Search bots and user bots read pages to answer a question, and the answer may link to you. So a company whose traffic to you is mostly training bots will always have a high ratio.
How the answers show links matters too. A platform that cites its sources prominently gives people more chances to click.
Working out your own
You need two numbers: bot visits from your server logs, and visits from AI assistants from your analytics. Put them side by side for each company.
In AnswerOrbit
Connect your server logs, and Bot visits in Website shows which AI bots visited and how often. Connect Google Analytics, and AI traffic shows the visits AI assistants sent you, by assistant. Together they give you both halves of the ratio for each company.