Crawlability
Crawlability is whether bots can reach your pages and read what's on them, which decides whether AI platforms can use those pages in answers.
Crawlability is whether bots can reach your pages and read what's on them. A page with good crawlability is allowed by your robots.txt, loads without errors, and has its content in the HTML your server sends.
For AI search, it's the first thing to get right. An AI platform can't name you because of a page it couldn't read.
What gets in the way
The common problems are simple, and easy to miss:
- Your robots.txt, or your firewall, blocks the bot
- The page returns an error to bots, even though it loads fine in a browser
- The text is written onto the page by a script after it loads, so a bot that only reads the HTML sees an empty page
- Important information sits behind a tab, a login or a click
The script problem catches a lot of modern sites. Google's crawler runs scripts, but many AI crawlers don't. A pricing page that loads its prices with JavaScript can look blank to them.
How to check
Open the page with scripts turned off, or look at the page source, and see whether the text you care about is there. Then check your robots.txt for the bots you want to allow, and ask whoever runs your hosting whether their bot protection blocks AI crawlers.
For example, a store might find that its product descriptions only appear after a script runs. Moving that text into the HTML the server sends makes it readable to every bot at once.
Crawlability in AnswerOrbit
Website shows whether each AI bot may read your site, and the Page tester checks a single page. The Pages tab lists your pages with their bot visits, and page checks show what a bot sees on each one. When a bot is blocked or a page hides its text from bots, Recommendations suggests the fix.