meta-externalfetcher
meta-externalfetcher is Meta's bot that fetches a single link when a person asks for it. Meta says it may not follow robots.txt rules.
meta-externalfetcher is the bot Meta uses to fetch one page when a person asks for it, for example when someone shares a link with Meta AI and asks what it says. Meta describes it as fetching "individual links at a user's request" in its list of web crawlers.
It works for one person at a time and only visits pages someone pointed it at. It doesn't roam the web on its own.
It may ignore robots.txt
Meta says that because this bot fetches pages at a person's request, it may bypass robots.txt rules. That's the main thing to know about it. A Disallow line in your robots.txt may not stop it.
Other AI companies say something similar about their user-requested bots, OpenAI among them on its bots page. The thinking is that the person could open the page in their own browser, so the bot is acting on their behalf. If you really need to keep it out, you'd have to block it at your server or firewall.
Should you worry about it?
Usually not. A visit from meta-externalfetcher means a real person was interested in your page and asked Meta's assistant about it. Blocking it means that person gets an answer without your page in it.
For example, someone in a WhatsApp chat pastes a link to Fernhill Coffee's subscription page and asks Meta AI "is this a good deal?" The fetcher reads the page so the assistant can answer. If it can't, the assistant has to guess, or say it can't open the link.
How to recognise it
Its user agent contains meta-externalfetcher/1.1. As with any bot, the name alone proves nothing, since anyone can borrow it. The request's IP address is the safer test.
In AnswerOrbit
The Audit tab in Website shows each AI bot we know with its purpose. Bots marked fetching for a user read a page because someone asked an assistant to look at it. See Website and AI traffic.