AhrefsBot
AhrefsBot is the crawler from the SEO tool Ahrefs, which reads pages and links to build Ahrefs' index and the Yep search engine.
AhrefsBot is the web crawler run by Ahrefs, the SEO tool. It visits pages across the web and records their content and the links between them. That data powers Ahrefs' reports, such as backlinks and rankings, and the Yep search engine, according to Ahrefs' page about its bots.
You'll see it in your server logs, often in large numbers. It's worth knowing what this bot does, so you can decide whether to let it in and so you don't mistake it for an AI bot.
What it's used for
When you or a competitor look up your site in Ahrefs, the links and pages you see were found by AhrefsBot. Ahrefs also runs a second bot, AhrefsSiteAudit, which crawls a site when its owner runs an audit in the tool.
Ahrefs says both bots follow robots.txt, allow and disallow rules alike, and respect a crawl delay. So you can control them the usual way:
User-agent: AhrefsBot
Crawl-delay: 10Should you block it?
Blocking AhrefsBot keeps your pages out of Ahrefs' index and Yep. It doesn't block Google, Bing or any AI assistant, which use bots of their own. It also doesn't hide your backlinks completely, because AhrefsBot finds many of them by crawling the sites that link to you.
For example, Kestrel Roasters notices AhrefsBot making thousands of requests a day to its product filters. Rather than block it, Kestrel disallows the filter paths for all bots, which cuts the noise and keeps its main pages in every index.
AhrefsBot and AI search
AhrefsBot isn't one of the AI platforms' bots, so letting it in or keeping it out won't change what ChatGPT or Perplexity say about you. If you're checking which bots can reach your site for AI search, look for the AI companies' own bots, like GPTBot, OAI-SearchBot and PerplexityBot.