An analyst asks an agent a question about a product category and gets back a clean answer: here are the strongest item affinities, here is what is different about these stores, here are two things you could do about it.
The answer reads well. It has real numbers in it. And the person holding it has exactly one question — is this good enough to act on?
Here is the thing I keep running into. That question is much harder than it looks, because a good answer and a bad answer look identical. Both are fluent. Both are specific. Both cite numbers. Fluency was never a signal of correctness. It just reads like one, and it reads like one to experienced people who know better.
So teams fall back on judgment. Someone senior reads it and says it looks about right. That works until the volume goes up. Once a digital twin agent is answering fifty questions a week instead of five, “someone senior reads it” is not a control. It is a bottleneck, and it quietly stops happening.
What checking it usually means today
The common answer is to have a second model grade the first one. Run the agent, then ask another model whether the answer was good. It is fast, it is cheap, and it produces a number you can put on a dashboard.
The problem is easiest to see with an analogy. Imagine grading a student exam by handing it to a second student — one who sat in the same lectures, read the same textbook, and revised from the same notes. When the two agree, you have not learned much. They were always going to make the same mistakes in the same places.
That is the situation with one language model grading another. They come from overlapping training data and share the same blind spots, so their mistakes line up rather than cancel out. The grader tends to approve exactly the kind of error it would have made itself. And because the grader is also probabilistic, its verdict is one more uncertain output. You have added a second estimate, not resolved the first one.
If an agent misreads the fiscal calendar and reports year to date as calendar year to date, a second model reading the same prompt will very likely make the same assumption and mark it correct. No amount of grading finds this. One query against the fiscal calendar table finds it immediately.
This is not an argument that LLM-as-a-judge is worthless. It is useful for style, tone, format, and obvious nonsense. It just cannot tell you whether a number is right, and that is usually the part you actually needed.
One vague question, three real ones
“Is this answer good?” does not have a single answer because it is not a single question. It is three, and they come apart in ways that matter. Split it, and each piece becomes something you can actually measure.
These three get blended into a single quality score all the time, and that is where the useful information gets lost. Look at what happens when you cross just the first two.
Row two is the one that matters most. An agent that is confidently wrong scores beautifully on any consistency-only or judge-based check: every run agrees, the writing is assured, a peer model signs off. It is the most likely failure to reach production, and repetition is precisely what makes it look credible. Only an outside anchor catches it.
Row three is the mirror image and gets under-rated. An answer that happened to be right once, through a path the agent will not repeat, is not a capability. It is a coincidence you cannot schedule.
Part 2 — the method, and why it is built this way
One constraint drives the whole design: every number the method reports has to come from something outside the model own probability space. That sounds austere, but it is just a discipline about where evidence comes from. Each of the three questions gets anchored to something you can point at.
Each of those is a measurement rather than an opinion, and none of them needs a second model to have a view about the first. Now let us take the three pillars one at a time, because the reasoning behind each technique is the part that usually goes unexplained.
Pillar 1 — accuracy, against something that is not a model
Before any trial runs, the claims the task is allowed to make are computed independently — a query against the warehouse, executed outside the agent and cached. When a run produces a number, it gets compared to that stored value. Match or no match.
There is no cleverness available here, and that is deliberate. Accuracy is the one lens where a probabilistic method would reintroduce exactly the problem we are trying to avoid. Any approach that infers correctness — a judge model, a heuristic, agreement between runs — is estimating rather than checking.
The cost is honest: someone has to write the ground-truth query. Teams adopt model-based grading specifically to skip this step, and the uncomfortable arithmetic is that it does not let you skip it. It relocates it. You still need labelled cases, now to calibrate the judge instead of to check the agent, and you need more of them, because the judge has to be calibrated across the whole range of inputs rather than just the ones you care about.
Pillar 2 — consistency, and why we shrink the estimate
The mechanic is simple: run the same task several times, each in a completely isolated session, then count how often each claim and each recommendation comes back. The complications are all in the details, and there are four worth naming.
First, what counts as the same answer. Two runs can express the same finding in different words, and a naive comparison treats them as two different findings. Get this wrong and every claim looks unique, which reads as total instability when the agent was being perfectly consistent. So claims are reduced to a canonical form before anything is counted — metric, value, scope, with the phrasing stripped out. This sounds like plumbing. It is the difference between a number that means something and a number that does not.
Second, small samples overstate their case. Suppose you run the task five times and a claim appears in all five. The raw frequency is 100%. Reporting that would mean promising a decision-maker that this claim always appears, on the strength of five observations.
So instead of raw frequency, the method uses a Bayesian estimate that pulls the number toward a neutral starting point by an amount that depends on how much evidence there is. Five unanimous runs report as about 86%, not 100%. Three unanimous runs report as 80% — same perfect agreement, less evidence, lower claim. This is standard practice anywhere estimates get made from small samples. It is the same reason a batter who goes 3-for-3 in his first game is not projected to hit 1.000.
The practical payoff is that the method can never report certainty it has not earned, and the only way to a tighter number is more trials. That is a knob the user controls, and the relationship between trials and confidence is explicit rather than hidden. The number that actually gets reported is the flip side: how likely is it that the next run says something different. A claim with a 14% chance of not reappearing is one thing. A claim with a 71% chance is another.
Third, a recommendation is not a claim. Claims are independent — each is present or absent. A recommendation is a choice among options that exclude each other, so it needs different treatment. Three votes to two is a lead, but it is not evidence of a settled preference, and reporting it as a decision would be overselling a coin flip. Treating the recommendation as a distribution over the available actions answers the real question: what is the probability this is genuinely the preferred action, rather than the one that happened to lead in this sample.
Fourth, “I do not know” is a good answer. A run that declines, that says the data does not support a conclusion, must not be counted as disagreement. Abstention is one of the more valuable behaviours an agent has, and an evaluation that penalises it will train it away. In one trial, four of five runs agreed exactly and the fifth abstained. That is a well-behaved agent. Any metric reporting it as 80% consistency is measuring the wrong thing.
Underneath all four sits a distinction worth making explicit, because it determines what you do next.
A single answer cannot tell these apart, and they need opposite fixes. “The agent got unlucky this time” and “the agent does not know this” look identical in one sample. Confuse them and you will rewrite prompts to fix randomness, or add trials to fix ignorance. Both burn time and neither works. Repeated sampling separates them cleanly: spread across runs is randomness, a tight cluster around the wrong answer is missing knowledge.
Pillar 3 — efficiency, read from the log
Every run leaves a trace: how many tool calls, how many repeated the same work, how many failed, how long it took. None of this needs interpreting. It is recorded.
The one piece of judgment is in the categories, and it earns its keep. A call blocked by policy is not the same as a call that errored. They look identical in a naive count and mean opposite things — a blocked call is a configuration problem you fix with a flag, an errored call is an agent problem you fix in the skill.
This came up immediately. An early run showed 64 failed tool calls, which looks like an agent that can barely function. All 64 were blocked by a permissions policy and zero were real errors. Separating the two categories turned an apparently broken fleet into a one-line policy change, after which the same task settled from 42–46 tool calls per run down to 15–20.
How this fits into real work
The usage pattern that emerged is not “run the evaluation suite before shipping.” It is closer to a second opinion, requested when you need one.
You ask a real question and get a real answer. You read it and you are not sure whether to act on it. So the first answer is distilled into a specification — the question, the scope, the metrics it claimed, the menu of actions it chose from — with every one of its own numbers removed. That specification is then run several times from a blank slate, and the original answer is located against the resulting pattern.
Two rules here are easy to get wrong. Strip the numbers: if the specification carries the first answer figures, every later run is anchored to them and you are measuring your own prompt. And never reuse the session: a run that can see the previous answer is influenced by it, and you will measure agreement you manufactured yourself. Carry the specification forward, never the conversation.
The output is not a pass or a fail. It is a position: this claim reproduced in four of five runs, this one appeared once, the recommendation held throughout, the number moved between 111 and 119. That is more useful than a score, because it tells you which part to trust and which part to check. The direction is settled, the precise figure is not — act on the recommendation, verify the number before it goes on a slide.
Part 3 — the mathematics (skip this if you do not need it)
Everything above reduces to a small amount of Bayesian arithmetic over counts. No model is consulted anywhere in the scoring path: the inputs are integers from the trial log, the outputs are posteriors with explicit intervals. Nothing in the final section depends on this one, so business readers can jump ahead without losing the argument.
For a declared metric slot, suppose a value appears in k of n valid runs. Treating each run as a Bernoulli draw with a uniform Beta(1,1) prior on the underlying rate gives a conjugate posterior.
That is Laplace rule of succession, and the shrinkage is the design rather than a side effect. With n = 5 and k = 5 the posterior mean is 6/7, about 0.857, not 1.0. Because the Beta-Binomial posterior predictive for the next draw is just the posterior mean, the probability that a fresh run fails to reproduce a claim has a closed form.
Notice the floor. At five runs a unanimous claim still carries a flip probability of 0.143, so a rule of “flip must be zero” is unsatisfiable by construction. Thresholds have to be set against the achievable floor for the trial budget. The bottom row is worth memorising: when every claim is seen exactly once, nothing matched, and you are looking at a broken matching function rather than an unstable agent.
Recommendations use a Dirichlet over the declared action menu, with dominance — the posterior probability that the leading action is genuinely modal — estimated by Monte Carlo over 10,000 draws with a fixed seed. Numeric claims use a Normal-Inverse-Gamma conjugate prior over unknown mean and variance, reported as a 95% predictive interval, because the useful question is the range a future run would land in rather than whether runs agreed. The two posterior quantities then combine into one categorical verdict with explicit, conservative thresholds.
Below the minimum trial count of three, the verdict is insufficient evidence. With no valid runs it is structural failure. A stable verdict is capped to borderline if any run failed, because a system cannot be called stable while evidence is being discarded. Efficiency is reported as the ratio of the most to the least expensive run, with redundant, blocked and errored calls held separately.
One last note from practice, because it generalises. The first live run returned a worst-case flip of 0.714 — that bottom row again. Claims were being compared as raw strings, so the same number phrased two ways counted as two claims, and 21 of the extracted claims were artefacts. Canonicalising claim identity dropped that to nine and brought flip to 0.14. The arithmetic was never wrong. A posterior computed over a broken identity function produces confident, well-formed nonsense.
Part 4 — what actually makes this different
None of the common alternatives is worthless. They answer narrower questions than teams tend to assume.
Against that backdrop, six choices define this approach, and each one came from something that went wrong in a real trial.
And one that is easy to miss: the evaluation carries the specification forward, never the conversation. Every trial starts from a blank slate with the first answer numbers stripped out. Small implementation detail, large consequence — if runs can see each other, you are measuring agreement you created, and the exercise reports your own assumptions back to you with a confidence interval attached.
Where this goes next
The efficiency lens is the least developed and probably the most interesting. Right now it measures cost. With enough runs across enough similar problems, the shortest correct paths become a training signal — not “this answer was good” but “this route was good,” which is a much more transferable thing to learn from.
That is the direction: from checking answers to improving the reasoning that produced them. Consistency tells you whether an agent has settled. Accuracy tells you whether it settled on the right thing. Efficiency is the one that can eventually tell it how to get there better.
Bottom line. You cannot establish that a probabilistic system is right by asking another probabilistic system. You can measure it against something fixed, observe how it behaves when you repeat it, and record what the journey cost. Three plain measurements, kept apart, reported with their uncertainty attached, and no model grading another model anywhere in the loop.