Mixed-use crawler
A mixed-use crawler is a bot that reads your pages for more than one purpose, such as search and AI, so blocking it affects both.
A mixed-use crawler is a bot that reads your pages for more than one job. One bot might collect pages for a search engine and for the company's AI answers at the same time, or for both search and training. Block it, and you block every job it does.
That's why blocking every bot that sounds like AI can cost you more than you meant. Some of those bots also keep you in search.
Examples
Googlebot is the clearest case. It crawls for Google Search, and AI Overviews and AI Mode are part of Google Search. Google gives you a separate token, Google-Extended, to control whether your content is used for its Gemini models, and says that token doesn't affect Search. It's all in Google's list of crawlers.
Meta's meta-externalagent is another. Meta says it crawls for "training foundation AI models or improving products by indexing content directly," two uses under one name.
Why it matters
When a bot does several jobs, blocking it means a trade. You might keep your pages out of training and also lose search traffic, or a spot in an AI answer, that you wanted.
For example, Harrow & Pine decides to block all AI training. Its web agency adds a long list of bots to robots.txt, Googlebot included, because it read that Google uses pages for AI. Two weeks later, the bakery's pages start dropping out of Google results. The fix is to unblock Googlebot and use Google-Extended to opt out of Gemini training instead.
How to handle them
Read each company's own page on its bots before you block anything. Look for the separate tokens many of them offer for training. Change one rule at a time, and watch what happens to your search traffic and AI visibility.
In AnswerOrbit
The Audit tab in Website shows each AI bot we know with its purpose, so you can see what you'd be blocking before you do it. If a bot that could read your site is suddenly blocked, we email you. See Website and AI traffic.