Log file analysis
Log file analysis is reading your server's records of every request to see how search engine and AI bots actually crawl your site.
Log file analysis is reading your web server's logs to see how bots move through your site. Every time anything requests a page, your server writes a line: the address asked for, the time, the response it gave and the name the visitor gave itself. Put those lines together and you can see which pages Googlebot, GPTBot or PerplexityBot read, how often, and where they hit errors.
It's the only way to see what bots actually did, instead of what you hope they did.
What you can learn from it
Logs answer questions no other report can:
- Which pages do bots read often, and which do they never visit?
- Which requests fail, and with what error?
- Do bots waste time on pages you don't care about, like filtered product lists or old campaign pages?
- After you changed robots.txt or launched a section, did the bots notice?
For example, say Harrow & Pine looks at a month of logs and finds that an AI search bot requested its product pages 3,000 times, and 600 of those requests got an error because a firewall rule treated the bot as an attack. Nothing in its analytics showed this, because bots don't show up in analytics.
Check who's really visiting
The name in a log line is only what the visitor claims. Anyone can call their scraper "Googlebot." So good log analysis checks each request against the addresses the company publishes for its bots, or looks the address up to see whether it belongs to that company. Google explains how in Verifying Googlebot, and the main AI companies publish their bots' addresses too.
Why it matters more for AI search
AI platforms run several bots each, for training, for search and for fetching a page someone asked about. Your logs are where you find out which of them read your site, and whether the search bots, the ones that feed answers, are getting through.
In AnswerOrbit
The Bot visits tab in Website does this for you once you connect your server logs. You can connect Cloudflare, Vercel, Amazon CloudFront, Google Cloud, Akamai, Fastly, Netlify or WordPress, send requests through a webhook, or upload an access log. We check every request before we count it, and show the ones that fail as Unverified requests.