X-Robots-Tag
X-Robots-Tag is an HTTP header that gives crawlers rules like noindex for a file, including PDFs and images that can't hold a meta tag.
X-Robots-Tag is a line your server can send along with a file, in the HTTP header, telling search engine crawlers what they may do with it. It takes the same rules as the robots meta tag, such as noindex, nofollow and nosnippet. The difference is where the rule lives: in the response from the server, not in the page's HTML.
That makes it the way to control files that have no HTML to put a meta tag in, like PDFs, images and spreadsheets. For a marketer, it's mostly something to know exists, so you can check for it when a page or file vanishes from search.
What it looks like
A server response might include:
X-Robots-Tag: noindex, nofollowYou can also aim a rule at one crawler by naming it, as in X-Robots-Tag: googlebot: noindex. Google lists the rules it supports in its robots meta tag documentation.
Your developer or host sets it, often with a rule that covers a whole folder or file type. That's handy, and it's also how mistakes spread. For example, Harrow & Pine adds a rule to keep its internal price lists out of search, and it accidentally catches every PDF on the site, including the care guides customers search for.
X-Robots-Tag and AI platforms
Since Google's AI Overviews and AI Mode use Google's index, a noindex in X-Robots-Tag keeps a file out of them too, and nosnippet limits what can be shown from it. Other AI platforms follow the rules their companies publish for their own bots, which are mostly about robots.txt. Check each company's page about its bots before relying on a header to keep content out of its answers.
Because the header is invisible on the page, it's easy to forget. When a page drops out of search for no clear reason, check the headers as well as the HTML.