Agent Output Verification Explained

A consultancy reviewing agent deployments this year describes a financial reconciliation agent that confirmed a transaction matched. It had hallucinated the matching record. The discrepancy wasn’t caught until month-end close.

We can’t independently verify that example, it’s an unattributed anecdote in a consultancy write-up, and we’d rather say so in a piece about agent output verification than pretend otherwise. But the example is worth paying heed to, because nothing in it resembles the failure most teams are guarding against.

The agent didn’t refuse. It didn’t produce obvious nonsense. It didn’t flag uncertainty. It completed its task, reported success, and was wrong in a way that surfaced weeks later, in a different process, run by different people.

That shape, confident, complete, and wrong, is what agent output verification is actually for. And most organizations look in the wrong direction.

Hallucinations are no longer the main event

Ask a room of executives what worries them about AI and you’ll hear “hallucinations.” That was the right fear in 2023.

It has aged. ChatSee, an AI reliability vendor, analysed a corpus of over 10,000 AI failure events assembled from public incident reports, community-reported failures, open-source agent traces and production observations between 2023 and 2026. In that corpus, fewer than 10% of failures were hallucination-related. The largest category, at 31.1%, was resolution and escalation breakdowns, the agent failing to finish properly or failing to hand off when it should have. Execution and workflow failures accounted for a further 19.4%, and were up roughly 62% against a Q2 2024 baseline.

Two caveats matter. This is vendor-published research behind an email gate, so the underlying data can’t be independently checked. And ChatSee states plainly that the figures “represent observed failure distribution within the analyzed corpus and should not be interpreted as absolute market incident rates.” It’s a curated collection, not a census.

Treat the numbers as directional. The direction is what matters here, and it’s consistent with what changed structurally.

A chatbot’s failure mode is saying something wrong, and you catch it because you’re reading the output. An agent’s failure mode is doing something wrong, calling a tool, updating a record, sending a message, marking a task complete, and you don’t catch it, because nobody is reading a transcript.

When output is generated and acted on faster than anyone reads it, the failure that matters isn’t the wrong sentence. It’s the wrong action, reported as a success.

So put a human on it, except humans have a specific blind spot

The obvious answer is human review. It runs into an uncomfortable, well-evidenced problem.

Researchers at Cohere and Oxford, in work presented at ICLR 2024, had crowdworkers and expert annotators evaluate the same model outputs. They then manipulated how assertive those outputs were, holding the task constant.

Crowdworkers consistently under-detected factual errors compared to experts. The size of that gap depended on tone. When outputs were written assertively, crowdworkers under-detected factual errors by 22.3 percentage points relative to experts. When outputs hedged and expressed uncertainty, the gap fell to 5.3 points.

Assertiveness also tracked closely with perceived quality overall, a correlation of 0.68, with confidently-worded outputs scoring 3.84 out of 5 against 2.95 for cautious ones.

The authors are careful about causation, and so should we be: they note it’s genuinely hard to tell whether assertive answers are simply better, or whether good answers read as more assertive. What the study does establish is narrower and still damaging: reviewers become substantially worse at catching factual errors as output gets more confident.

Now hold that next to what an LLM produces by default. Fluent, structured, confident prose, whether or not the content beneath it is right. Hedging is precisely what these systems have been trained out of.

The reviewer’s blind spot and the generator’s default style point the same direction.

This isn’t an argument against human review. It’s an argument against human review as the only control, performed casually, by someone with thirty seconds and no rubric.

So use an LLM to check the LLM, that has its own problems. The other obvious answer is LLM-as-judge. It scales, it’s cheap, it’s consistent, and it’s now standard practice. It’s also biased in documented ways. Most of what we know traces to one careful paper, Zheng et al., presented at NeurIPS 2023, which built MT-Bench and Chatbot Arena specifically to test whether LLMs can judge.

– Position bias is real and can be severe. In their tests, the worst judge-and-prompt combination favoured whichever answer was presented first in 75% of cases. That’s a worst case on 2023-era models in a deliberately adversarial setup, so don’t read it as typical. Later work across 150,000+ judgements found roughly a quarter of verdicts flip when you swap the order, less dramatic but still enough to matter.
– Verbosity bias. Longer answers score higher. Stronger judges resist this partially, not fully.
– Self-preference bias is likely, though weaker than often claimed. Zheng et al. observed GPT-4 favouring its own outputs by 10% and Claude by 25%, then explicitly declined to conclude that self-enhancement bias existed, citing limited data. Later work established the effect more firmly by linking it to self-recognition. Worth knowing before you have a model grade its own family’s work.

Here’s the finding that reframes all of it. In the same study, GPT-4’s agreement with human evaluators reached 85%, higher than the agreement between humans, which was 81%.

You can read that two ways and both are true. LLM judges are roughly as reliable as people. And people agree with each other only about four times in five.

(Those are the paper’s headline figures, which exclude tied votes. Counting ties, agreement drops to 66% for the model and 63% between humans, lower, and still pointing the same way.)

Neither reviewer is a foundation. That’s the actual point.

Agent output verification is a system, not a checkpoint

If both available reviewers are biased, and one of them is the same technology that produced the work, the answer isn’t picking the better reviewer. It’s designing verification as layers that fail differently.

Four things make the difference.

Verify against the outcome, not the task. The one most organizations skip, and the one that would have caught the reconciliation error. “Did the agent complete the reconciliation?” gets a yes. “Does the reconciled balance match the source system?” gets a no. The first question checks the agent’s own account of its work. Only the second checks reality.

Check the premise and not just the reasoning. A large share of what looks like reasoning failure is a context failure, the agent reasoned correctly from information that was wrong, missing, or invented. Verifying the inputs an agent acted on is often cheaper and more revealing than auditing its conclusions.

Use human and LLM review for different jobs. LLM review is good at breadth: scanning everything, applying consistent criteria, flagging anomalies at volumes no team could read. Human review is good at judgement on the flagged subset, and on questions a rubric can’t anticipate. Using either for the other’s job is where teams get burned. And given self-preference, keep the judge independent of the generator where you can.

Apply explicit rubrics rather than impressions. One 2026 preprint found that structured, multi-dimensional evaluation criteria reduced self-preference bias by around 31% on average, though that’s the authors testing their own proposed fix, so take it with a grain of salt. The underlying logic is sound and applies to human reviewers too: a rubric forces attention onto dimensions the reviewer would otherwise skip in favour of “does this read well.” It does the work that fluency would otherwise do, which, per the ICLR finding above, is exactly the failure you’re trying to prevent.

The uncomfortable economics

Verification is a cost centre until the moment it isn’t.

Every layer adds latency and expense to work that already felt finished. There’s constant pressure to skip it, and skipping it is invisible, right up until a reconciliation error surfaces at month-end, or a number reaches a board pack that nobody can trace.

The trap is that unverified agent output looks like productivity – completed tickets, merged code, closed tasks. Every dashboard turns green. The cost of skipping verification doesn’t appear anywhere you’d think to look, which is precisely why teams keep skipping it.

Unverified agent output is just faster technical debt, and some of it surfaces in an audit rather than a bug tracker.

What to ask on Monday

You don’t need an evaluation framework to start. You need one question, asked about a real piece of agent work that shipped last week:

“How would we know if this had been wrong?”

If the answer is “someone would have noticed”, ask who, and when, and whether they were looking. If it involves a person, ask what they compared it against. If it’s “we’d have found out from the customer,” you have your finding.

The organizations getting real value from agents are the ones that can answer that question about work they’ve already shipped.

The agent finishing is not evidence. The outcome moving is.

—

Rezoomex gives teams a way to assemble context deliberately, observe what agents actually produce, run layered human and LLM review, and apply safety rubrics before anything ships.

Next in this series: how payment is changing in the agentic economy, and why outcome-based pricing only works if you can verify the outcome first.

—

References

Berton, L. (2026, June 1). AI agent production failures: Enterprise lessons from 2026’s first wave. Open Empower. https://www.openempower.com/blog/ai-agent-production-failures-enterprise-lessons-2026

ChatSee. (2026, July 29). New research finds enterprise AI failures are shifting beyond hallucinations as companies move from chatbots to agents [Press release]. PR Newswire. https://www.prnewswire.com/news-releases/new-research-finds-enterprise-ai-failures-are-shifting-beyond-hallucinations-as-companies-move-from-chatbots-to-agents-302837907.html

Hosking, T., Blunsom, P., & Bartolo, M. (2024). Human feedback is not gold standard. International Conference on Learning Representations (ICLR 2024). arXiv:2309.16349. https://arxiv.org/abs/2309.16349

Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. NeurIPS 2024. arXiv:2404.13076. https://arxiv.org/abs/2404.13076

Yang, et al. (2026). Quantifying and mitigating self-preference bias of LLM judges [Preprint]. arXiv:2604.22891. https://arxiv.org/abs/2604.22891

Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arXiv:2306.05685. https://arxiv.org/abs/2306.05685

—

A note on sources. The evaluator-bias findings (Hosking et al., Zheng et al., Panickssery et al.) are peer-reviewed and are the most reliable evidence in this piece. The rubric-mitigation figure is an unreviewed preprint. The failure-distribution statistics are vendor research on a curated corpus, and the two opening anecdotes are unattributed examples from a consultancy blog, included as illustration, not evidence. Where these disagree, trust them in that order.