CCBot
CCBot is Common Crawl's crawler. It builds a free, public archive of the web that many AI models have been trained on.
CCBot is the crawler run by Common Crawl, a non-profit that keeps a free, open archive of the web. Anyone can download the archive and use it, and AI researchers and companies have used it widely to build training data for their models.
That makes CCBot different from the bots of any one AI company. Letting it in, or keeping it out, decides whether your pages end up in a dataset that many other organizations draw on.
Why one crawler reaches so many models
Training a language model takes an enormous amount of text, and crawling the whole web yourself is expensive. Common Crawl's archive offers a ready-made copy, so many teams start from it. OpenAI's GPT-3, for example, was trained largely on a filtered version of it, as its paper describes.
So when CCBot reads your page, the page may end up in the training data of models you've never heard of, built by teams you'll never deal with.
Blocking it
If you'd rather your pages weren't in the archive, Common Crawl says to add this to your robots.txt:
User-agent: CCBot
Disallow: /Think about what you're trading. Blocking CCBot keeps future crawls of your pages out of the archive. It doesn't remove what's already been collected, and it has nothing to do with the bots AI assistants use to search the web and cite pages in their answers. Those have their own names, and need their own rules.
For example, Harrow & Pine decides it doesn't want its product descriptions used for training. It blocks CCBot and the AI companies' training bots, and leaves their search bots allowed, so assistants can still find and cite its pages.
Spotting the real one
Common Crawl warns that other crawlers sometimes call themselves CCBot. The real one comes from addresses Common Crawl publishes, and its address looks up to a Common Crawl host name.
In AnswerOrbit
The Audit tab in Website shows whether your robots.txt allows each of the AI bots we know, and the Bot visits tab shows which ones actually came, once you connect your server logs. See Website and AI traffic.