Amazonbot
Amazonbot is Amazon's web crawler. It reads pages to improve Amazon's products and services, and its data may be used to train Amazon's AI models.
Amazonbot is Amazon's main web crawler. It visits websites and reads their pages to improve Amazon's products and services. Amazon says the data it collects may also be used to train Amazon's AI models. Its page for site owners describes it, along with Amazon's two other bots.
Amazon's three bots
Like OpenAI and Anthropic, Amazon now runs separate bots for separate jobs:
- Amazonbot, the general crawler, which may feed AI training
- Amzn-SearchBot, which reads pages for search in Amazon products like Alexa, and not for training
- Amzn-User, which fetches a page when a person's request needs fresh information
That split lets you choose. You can block Amazonbot to keep your pages out of training, and still let Amzn-SearchBot in so you can appear in Amazon's answers.
How to control it
Amazon says Amazonbot follows robots.txt, including allow and disallow rules for its name. It doesn't support the crawl-delay rule. It also reads page-level robots meta tags, and treats noarchive as a request not to use that page for model training.
For example, Harrow & Pine's robots.txt could keep Amazonbot away from its whole site:
User-agent: Amazonbot
Disallow: /Amazon notes changes can take about a day to take effect. Anyone can call their bot Amazonbot, so Amazon publishes the addresses its crawler uses, which lets you check a visit is real.
Amazonbot in AnswerOrbit
The Audit tab in Website reads your robots.txt every day and shows, for each AI bot we know, whether it's Allowed, Partly allowed or Blocked. Connect your server logs and the Bot visits tab shows which AI bots visited and which pages they read. We only count a visit when its address is on the company's published list or looks up to its own host name.