Primer / 2 of 4
What we mean when we say hallucination
Not a bug you patch. A consequence you measure, and an incentive you can stop paying.
The word does a lot of damage. “Hallucination” sounds like a malfunction — a component misfiring, something a patch will fix. It is the opposite. It is the model doing exactly what it was trained to do, in a situation where the training taught it a bad habit.
That distinction is not pedantry. It decides whether your requirement reads “the agent must not hallucinate”, which is unmeasurable and unachievable, or “95% of factual claims must resolve to retrieved evidence”, which is neither. This primer is about how to get to the second one. It follows on from the first, where the loop that causes all of this is set out in six steps.
Part 1 / Origin
Where it comes from, and why scale will not remove it
Published analysis puts this in two stages, and the two have different remedies. The first is baked in during pretraining. The second is kept alive afterwards, by us.
The origin: prediction over unlabelled text
Pretraining is next-word prediction over an enormous corpus. That corpus carries no labels. Unlike a folder of cat and dog photos, it contains only positive examples of fluent language — there is no tag anywhere saying “this claim was false”.
So validity has to be inferred rather than learned. For arbitrary low-frequency facts, that inference cannot be made reliable by more data.
The persistence: grading that rewards the guess
Post-training tries to squeeze hallucination out. Evaluation puts it back, because the dominant scoreboards score accuracy and nothing else.
Under accuracy-only scoring, a guess strictly dominates an abstention. Do that across thousands of benchmark questions and the model that guesses tops the table.
Why spelling errors disappeared and birthdays did not
Grammar and spelling follow consistent patterns, so more data and more parameters drive those errors towards zero. Arbitrary facts do not follow a pattern. Nothing about the shape of a name predicts a date of birth, so no amount of scale lets a next-token objective derive one it never saw often enough.
The analogy in the paper is a good one: imagine training a classifier to label pet photos by the animal’s birthday. However good the algorithm gets, the error rate does not fall, because the label is not a function of the input. The formal result the authors derive is in the same spirit — the rate at which a generative model produces invalid statements is bounded below by roughly twice the rate at which it would misclassify a statement’s validity.
Scales away
Errors with a pattern behind them
Spelling, grammar, formatting, syntax, common phrasing. The signal is everywhere in the corpus and consistent, so scale takes the error rate down.
Does not scale away
Arbitrary, low-frequency facts
Dates, identifiers, citations, prices, one-off attributions, anything seen once or never. No pattern to generalise from, so statistical pressure produces a plausible answer instead of no answer.
Part 2 / Incentive
The scoreboard is teaching it to guess
Think of a multiple-choice exam where a blank scores zero and a guess has, say, a one-in-365 chance of paying off. There is no state of knowledge in which leaving it blank is the better play. That is the incentive structure almost every public benchmark presents, and models are optimised against public benchmarks.
What it looks like in numbers is stark. Here are two models from the same vendor on SimpleQA, a factuality benchmark, as reported in the GPT‑5 system card.
| Metric | gpt‑5‑thinking‑mini | o4‑mini |
|---|---|---|
| Abstention rate — said it did not know | 52% | 1% |
| Accuracy — the headline number | 22% | 24% |
| Error rate — confidently wrong | 26% | 75% |
The second model wins on the metric that gets published, by two points, while producing a wrong answer three times as often. A user interacting with it is wrong three quarters of the time rather than a quarter of the time. Ranked on accuracy, that is an improvement.
Where your organisation enters the story
You will recreate this internally without meaning to. If your acceptance criteria grade an agent on task completion, and “I could not determine that from the records available” is logged as a failure, you have built the same incentive into your own programme — then tuned prompts, retrieval and routing until the guessing behaviour won.
Evaluation design inside your company shapes honesty inside your product. Arguably more than vendor selection does.
Part 3 / Definition
A definition you can actually test against
Ours, and the one we recommend adopting in writing:
Working definition
A hallucination is confident output that is not supported by the evidence available to the system — produced by the same mechanism that produces the correct answers.
Two things fall out of that wording, and both matter when you write the test.
There is no fabrication module to switch off. The correct answer and the invented one come out of the same loop, at the same cost, with the same fluency. Fluency is a stylistic property and it is close to independent of correctness, which is why a review process that reads the output and forms an impression will never catch this reliably.
A statement can be true and still unsupported. The model may produce a fact that happens to be right but that nothing in the retrieved evidence establishes. For assurance purposes, unsupported is the thing you police. You cannot audit luck, and a system that is right by coincidence today has no property that will hold tomorrow.
Two analogies that hold up with technical audiences
- A curve fit with no error bars. Ask it for a value in a region where it never saw data and it returns one anyway, with the same number of decimal places as everywhere else. It has no way to say “off support”, because nothing in the fit represents support.
- The exam candidate who never leaves a blank. Not dishonest — correctly optimised for the marking scheme in front of them. Change the marking scheme and the behaviour changes with it.
Part 4 / Controls
What to require instead
“The agent must not hallucinate” fails the first test of a requirement: nobody can tell you whether it has been met. Every row below is something a tester can run and a reviewer can sign.
| Instead of | Write |
|---|---|
| “The agent must not hallucinate.” | Groundedness ≥ 0.95 on the curated evaluation set: every factual claim resolves to a span in a retrieved source. |
| “Answers must be accurate.” | Accuracy on the golden set reported alongside abstention rate and error rate. All three, always together. |
| “It should say when it is unsure.” | On the unanswerable subset — cases with no supporting evidence in the corpus — the agent abstains or escalates in at least 95% of runs. |
| “Cite sources.” | Every citation resolves to a document that exists and contains the claim. Broken or non-supporting citations are counted and fail the gate. |
| “Get the numbers right.” | Arithmetic is recomputed deterministically outside the model and compared. A mismatch blocks the action rather than annotating it. |
And four habits behind those numbers
- Build an unanswerable set on purpose. A golden set made only of answerable questions cannot detect the failure mode in this whole article. Ten to twenty per cent of cases should have no supported answer, and the correct behaviour on them is to decline.
- Make honest uncertainty a pass, in the rubric. Confidently wrong should score worse than “I could not determine that”. Where the stakes allow it, give the agent an explicit “flag for review” path and count using it as success, not as containment lost.
- Do not reach for temperature zero and call it fixed. Repeatability is not correctness — a deterministic wrong answer is still wrong — and provider-side non-determinism erodes the guarantee anyway.
- Keep a failure taxonomy. Unsupported claim, wrong entity, stale record, misread evidence, right answer for the wrong reason. Counts by category tell you which control to build; a single “hallucination rate” tells you nothing you can act on.
The reframe is the whole point. Hallucination is not a defect report to be escalated to a vendor. It is a rate — one you can measure, budget for, and hold below a threshold you have agreed with the business, using controls you own.
Sources
- OpenAI and Georgia Tech — Why language models hallucinate, Kalai, Nachum, Vempala and Zhang (openai.com)
- OpenAI — GPT‑5 system card, for the SimpleQA abstention and error rates (openai.com)
- Anthropic — On the biology of a large language model, for the internal circuitry behind refusal and recall (transformer-circuits.pub)
- NotedML — Agentic AI Assurance, domain D1, for where groundedness sits in the wider framework
Next in the series
The model can explain itself. That is a different question from whether the explanation is true.
Primer 3 goes at the reasoning trace: what published experiments found when researchers checked whether a model’s stated reasoning matched what actually drove its answer, and what that leaves you able to claim in an audit.