LLM-as-a-judge
LLM-as-a-judge means using one AI model to grade or label another model's output, such as whether an answer is positive about a brand.
LLM-as-a-judge means using one AI model to grade or label the output of another. You give the judge model an answer and a question about it, like "Is this answer positive about Fernhill Coffee?" or "Does this sentence match the fact that delivery is free?," and it gives you a verdict.
It's how many AI search tools read answers at scale. Knowing that helps you read their numbers with the right amount of trust.
Why tools use it
Reading thousands of answers by hand isn't practical. A model can label tone, spot claims, or decide whether a page supports a statement much faster. The approach was studied in a 2023 paper, which found strong models often agree with human graders, though not always.
Where it goes wrong
A judge model makes mistakes like any model. It can be swayed by long answers, by the order things appear in, or by confident wording. It can disagree with itself if you ask twice.
For example, an answer says "Kestrel Roasters is pricier than most, but the beans are worth it." One judge might call that positive, another neutral. A sentiment score built from thousands of verdicts like that can move without anything real changing.
How to read a judged number
Ask the tool what each verdict is based on, and whether you can see it. You can check a number when you can click through to the sentence behind it. A score with no way to see the reasoning isn't.
In AnswerOrbit
We show our working. Sentiment comes with the words each verdict is based on, and we count positive, neutral and negative mentions without turning them into a score. We only say a claim contradicts one of your facts when we can quote the sentence that says it. If you think we got a claim wrong, click Change verdict. See How we count and Claims and facts.