AnswerOrbitDocs
Glossary
Glossary

Indexing

Indexing is when a search engine stores a page it has crawled, so it can find that page again and show it, or hand it to an AI answer.

Indexing is when a search engine stores a page it has read, so it can find it later. A crawler visits the page, the search engine works out what it's about, and the page goes into the index: a huge catalogue the search engine looks things up in.

It matters for AI search because many AI answers start with a search. If your page isn't in the index the platform searches, it can't be found, read or cited.

Crawled, indexed, ranked

These are three different steps. Crawled means a bot fetched the page. Indexed means the search engine decided to keep it. Ranked means it comes up for a given search. A page can be crawled and never indexed, for example if it's a near copy of another page or has a noindex tag.

Google explains the steps in how Search works.

Which index does each platform use?

Each AI platform searches an index, and they aren't all the same one. Google AI Overviews and AI Mode are part of Google Search, so they draw on Google's index. Copilot is Microsoft's, and searches with Bing. Some AI companies also run search bots of their own that build their own index.

For example, Harrow & Pine publishes a new page on its spring menu. Google indexes it in two days, and it starts showing up in AI Overviews. Bing hasn't picked it up yet, so Copilot still describes the winter menu. Checking Bing Webmaster Tools tells them why.

How to help your pages get indexed

Let search bots read the page in your robots.txt, put the text in the HTML your server sends, link to the page from other pages on your site, and list it in your sitemap. You can ask Google and Bing to look at a page sooner from their webmaster tools.

In AnswerOrbit

The Audit tab in Website shows which AI bots your robots.txt lets in, and the Pages tab shows what a bot reads on each page. See Website and AI traffic.

On this page