AI2Bot
AI2Bot is the web crawler of Ai2, the Allen Institute for AI, which collects pages to train its open language models.
AI2Bot is the web crawler run by Ai2, the Allen Institute for AI, a nonprofit research lab. It visits websites and collects pages, which Ai2 says it uses to train open language models. Its crawler page explains what the bot does and how to get in touch.
It's a training bot. It doesn't fetch pages for a chat assistant to cite in an answer, so blocking it doesn't keep you out of ChatGPT, Perplexity or Google.
How to spot it
AI2Bot names itself in every request. In your server logs, its user agent reads:
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)Ai2's page suggests using that name to filter its visits.
Allow it or block it?
That's your call, and it depends on how you feel about your pages training AI models. Some brands are happy for their content to feed open research. Others prefer to keep it out of every training set.
If you want to block it, add a rule to your robots.txt for the name AI2Bot. Ai2's page doesn't say whether the bot follows robots.txt, so if it matters to you, check your server logs afterwards, or block it at your host or CDN.
For example, Fernhill Coffee might choose to block every training bot, AI2Bot included, while letting in the search bots that help it get named in answers.
AI2Bot in AnswerOrbit
The Audit tab in Website reads your robots.txt every day and shows, for each AI bot we know, whether it's Allowed, Partly allowed or Blocked, along with its purpose. Connect your server logs and the Bot visits tab shows which AI bots actually visited, and which pages they read.