User agent
A user agent is the name a bot or browser gives with every request to your site, like GPTBot. robots.txt rules are written against it.
A user agent is the name a bot or browser sends with every request it makes to your website. OpenAI's training bot calls itself GPTBot. Anthropic's calls itself ClaudeBot. Your own browser sends one too, saying which browser and device you're on.
It's the handle you use to tell AI bots apart. Every rule in your robots.txt starts with a user agent, so knowing the names is how you let one bot in and keep another out.
Where you'll see it
You'll meet user agents in two places. In your robots.txt, each group of rules starts with a line like User-agent: GPTBot. And in your server logs, every request is listed with the user agent that made it.
Each AI company lists the names of its bots and what each one is for: OpenAI, Google, Anthropic and Perplexity. Most run several, one for training, one for search, and one that fetches a page when a person asks.
An example
For example, Harrow & Pine wants its pages in ChatGPT's answers but out of OpenAI's training. Its robots.txt blocks GPTBot and allows OAI-SearchBot. Both belong to OpenAI. The user agent is what lets the file treat them differently.
A name is only a claim
Anyone can put any name in their requests. A scraper can call itself GPTBot. So a user agent only tells you who a visitor says it is. To check, you compare the visitor's address with the addresses the company publishes for its bot, or look up the address's host name. Newer methods like Web Bot Auth let bots sign their requests instead.
In AnswerOrbit
The Audit tab in Website and AI traffic reads your robots.txt and shows, for each of the 42 AI bots we know from 25 companies, whether it's allowed, and whether its own rule or your file's general rule decides it. In Bot visits, we check every request against the company's published addresses or host name before we count it. Requests that fail are shown as Unverified requests.