AnswerOrbitDocs
Glossary
Glossary

Pay-per-crawl

Pay-per-crawl is charging AI companies a fee each time their crawler reads a page on your site.

Pay-per-crawl is a way to charge AI companies each time their crawler reads one of your pages. The site sets a price, and a crawler that wants the page either agrees to pay or doesn't get it.

It matters because, until recently, a site had two choices with an AI crawler: let it in for free, or block it. Pay-per-crawl adds a third. For publishers whose pages are read far more by bots than by people, it's a way to be paid for that reading.

How it works

The best-known version is Cloudflare's. Its pay per crawl feature, part of AI Crawl Control, lets a site owner set a price for their site. A crawler that sends a request saying it will pay gets the page as usual. A crawler that doesn't gets an HTTP 402 Payment Required response with the price. Cloudflare handles the payment between the two. At the time of writing, Cloudflare lists it as a closed beta.

The 402 status code has been in the HTTP standard for years with no common use. Pay-per-crawl is one of the first real uses for it.

What to weigh before you charge

Charging for crawls has a cost of its own. The bots that read pages for answers, the ones that let a platform cite you, are the same kind of bot you'd be charging. If a platform's search bot won't pay, it won't read your page, and it can't name or link to you.

For example, a trade magazine that earns from advertising might charge training bots and let search bots in free, so it keeps getting cited. A brand like Fernhill Coffee, which wants to be named in as many answers as possible, has little reason to charge at all.

So decide which bots you'd charge, and what you'd lose if they walk away.

In AnswerOrbit

The Audit tab in Website shows each AI bot's purpose (training, search, or fetching for a user) and whether your robots.txt lets it in. That's the list to start from when deciding which bots to charge, block or welcome.

On this page