Non-determinism
Non-determinism means an AI platform can give different answers to the same question, even when nothing else has changed.
Non-determinism means an AI platform can give different answers to the same question. Ask ChatGPT for the best coffee subscriptions on Monday, ask again on Tuesday with the same words, and you may get a different list of brands.
For a marketer, this changes how you read any result. One answer is one sample, a bit like a single roll of the dice. It tells you what can happen, not what usually happens.
Why answers change
A language model writes an answer one piece at a time, and at each step it picks from several likely next words. That choice has some chance in it, so two runs can drift apart early and end up naming different brands.
On top of that, the platform itself changes things from one question to the next. It may search the web on one run and not the next, run different searches, read different pages, or answer with a different model. Any of those can change which brands make it into the answer.
What it means for measuring
Because answers vary, you need many of them before a number means anything. A single check in which you weren't named doesn't tell you much. Fifty checks in which you were named twice does.
For example, say Kestrel Roasters asks one prompt once a day. In a week, it's named on Monday, Wednesday, Thursday and Sunday. Any single day would have told Kestrel it was always named, or never named. The week says four times out of seven.
It also means small changes from one week to the next can be luck. Before you act on a drop, check that it's bigger than the normal ups and downs.
Non-determinism in AnswerOrbit
We ask each prompt once a day on each platform and base every number on the last seven days of answers, with the number of answers shown next to it. We only mark a change when luck alone would make a difference that big less than 5 times in 100. How we count explains the test, and Your first days explains why changes wait a week.