meta-webindexer
meta-webindexer is Meta's crawler for Meta AI's search results. Letting it read your site helps Meta AI find your pages when it answers.
meta-webindexer is the crawler Meta uses to improve the search results inside Meta AI. In its list of web crawlers, Meta says it "navigates the web to improve Meta AI search result quality for users."
So it's the Meta bot that decides whether your pages can turn up when Meta AI searches. If you want to be found in Meta AI's answers, this is the one to let in.
How it differs from Meta's other bots
Meta runs several bots, each with its own name, so you can treat them differently in robots.txt:
- meta-externalagent collects pages to train AI models and to index content for Meta's products
- meta-webindexer reads pages for Meta AI's search
- meta-externalfetcher fetches a page when a person asks for it
This split lets you keep your content out of training without dropping out of Meta AI's search, or the other way round.
How to allow it and block training
For example, Kestrel Roasters wants Meta AI to find its brewing guides, but doesn't want them used to train models. Its robots.txt could say:
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalagent
Disallow: /Meta says robots.txt changes can take up to 24 hours to apply, because its crawlers may keep a copy of the file for that long.
How to recognise it
Its user agent contains meta-webindexer/1.1. A request using that name isn't proof it came from Meta, since anyone can borrow a bot's name. The request's IP address is the safer test.
In AnswerOrbit
The Audit tab in Website reads your robots.txt every day and shows, for each AI bot we know, its purpose and whether it's allowed, partly allowed or blocked. If a bot that could read your site is suddenly blocked, we email you. See Website and AI traffic.