Five frontier LLMs disagree on 67% of real-world fact-checks: what it means for building on AI

2026-06-02 · ~6 min read · Conor Dobbs

A research group called Lenz ran a study that should change how you think about trusting an AI's factual answers. They took 1,000 recent real-world claims that actual users had submitted to a fact-checking platform, handed each claim to five frontier models, and asked every model for a verdict. On 67% of those claims, at least one model dissented from the panel majority. On 34%, the gap between the two most-disagreeing models was two full buckets or more. That is not a calibration wobble. That is two top-tier models looking at the same claim and landing on opposite sides.

The five models were GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro with Search, and Sonar Pro. Each one picked from four verdicts: True, Mostly True, Misleading, False. No explanations, no hedging, only the label. The study did not score anyone against a ground truth, so this is not a leaderboard of who is right. It is a measure of how often the best models we have today fail to agree on what is true.

Why this study is different from a benchmark

Most model comparisons run on benchmarks with public answer keys, which means the questions have probably leaked into training data and the answers are clean and settled. This corpus was the opposite. These were fresh claims that real people cared enough about to submit for checking, spanning science, healthcare, politics, and law. The models had not seen them. There was no tidy right answer waiting at the back of the book.

That is closer to how any of us really use these tools. You do not ask a model for the capital of France. You ask it whether a specific drug interaction is real, whether a clause in a contract means what you think, whether a news claim from last week holds up. Messy, recent, consequential. And that is exactly where the study found the models splitting apart.

The shape of the disagreement is the interesting part. The panel converged on the definitive ends of the scale. When something was clearly True or clearly False, the models tended to line up. The fracture happened in the middle, on Mostly True and Misleading. There were zero unanimous "Mostly True" verdicts across all 1,000 claims. The nuance, the "it depends," the partial truth, is precisely where the models stopped agreeing with each other. Krippendorff's alpha came out at 0.639, below the 0.8 that researchers usually treat as the floor for reliable agreement. Peer agreement between any two models ranged from 53% to 75%.

The honest take

It would be easy to turn this into an "AI is broken" headline. That is not the lesson, and I do not buy it. I run autonomous coding agents every day and I maintain a small site comparing the tools I use, so I have no interest in pretending these models are worse than they are. They agree on the clear cases. The disagreement clusters in genuinely ambiguous territory where two careful humans would also split.

The lesson is narrower and more useful: a single model's verdict on a non-obvious factual claim is one opinion, not a settled answer. We have spent two years training ourselves to read a confident model response as if it were a lookup. This study is a clean reminder that on the claims that matter, swap the model and you can get the opposite verdict, with the same confident tone attached.

That reframes a few habits worth changing.

If a factual claim is consequential, do not let one model be the final word. The cost of a second opinion is one more API call. The study makes the case empirically: on a third of these claims, a second model would have moved you two buckets. If you are shipping anything where a wrong "True" has real downside, fan the claim out to more than one model and treat divergence as a flag to slow down and check a source, not as noise to average away.

Retrieval helps less than you would hope. Two of the five models had live search, and the panel still fractured. Grounding a model in documents narrows some gaps, but it does not deliver consensus on ambiguous real-world claims. Do not assume that bolting on search turns a verdict into a fact.

The disagreement is worst where the stakes are highest. The middle of the rubric is health nuance, legal interpretation, political claims with partial truth. Those are the exact domains where you least want to trust a lone confident answer, and they are where the models are least reliable. The risk and the unreliability point the same direction.

What this means for coding agents

I mostly use these models to write and operate software, and the same principle carries straight over. When an agent tells you "this library handles retries automatically" or "that flag is safe to set in production," it is issuing a verdict, not reading from a manual. It is the same act of confident judgment the fact-check study measured, and it fails the same way on the non-obvious cases. The fix is the one good engineers already use: verify, do not trust. Run the code. Read the actual docs. Check the claim against a source before you build on it. The models are excellent at producing a plausible answer fast, and that is genuinely valuable, but plausible and verified are different things, and the gap is widest exactly where it costs the most.

I have written before about the tells that give away AI-generated writing and code and about a real case where an AI report shipped with hallucinated citations. This study is the quantified version of the same warning. The confidence in the output is not evidence about the truth of the output.

The caveat that keeps it honest

Disagreement is not the same as error. The study measured how often models diverge, not which one was wrong, because there was no ground truth to grade against. It is possible that on many of these claims one model is reliably right and the others are reliably wrong, and we cannot tell from divergence alone. An alpha of 0.639 is limited agreement, not random noise. And again, on clear-cut claims the panel mostly agreed.

But for the practical question of whether you can take one frontier model's factual verdict at face value, the answer the data gives is no. Not on the claims that are worth checking. Treat a single model's "True" as a strong hypothesis, not a citation, and design whatever you are building so that the consequential claims get a second look.