Answer variability
Answer variability is how AI platforms give different answers to the same question, even when nothing has changed. It's why one check proves little.
Answer variability is the way AI platforms give different answers to the same question. Ask ChatGPT for the best coffee subscription twice, a minute apart, and you can get two different lists of brands. Nothing changed about the brands. The platform just answered differently.
It's the most important thing to understand before you measure AI answers, because it means a single check tells you very little.
Why answers vary
A few things go into it:
- The model picks its words with an element of chance, so two answers to the same question can differ
- The platform may run different web searches, read different pages, or skip the search altogether
- Pages change, so what it finds today isn't what it found yesterday
- The platform updates its models and settings from time to time
- When you're signed in, it knows your location and past chats, and adjusts to them
What this means for measuring
Think of it like weighing yourself. Your weight moves a little each morning, so one reading tells you almost nothing. A week of readings tells you where you are.
The same goes for AI answers. For example, say Fernhill Coffee checks ChatGPT once on Monday and isn't named. On Tuesday it's named first. Neither day means much. Asked every day for a week across 10 prompts, though, Fernhill gets 70 answers, and the share of those that name Fernhill is a number worth reading.
So be careful with screenshots. One answer showing you at the top, or missing, is one roll of the dice.
In AnswerOrbit
We ask each of your prompts once a day on each platform, as someone who isn't signed in, and every number is based on the last seven days of answers. Each number also shows how many answers it's based on, so you know how much to trust it. Answers explains why ChatGPT can show you something different when you ask it yourself.