Primer / 2 of 4
What we mean when we say hallucination
Why models make things up, how our own scoring rewards it, and what to write in a requirement instead.
The word gets in the way. “Hallucination” sounds like a broken part. Something misfires, someone ships a patch, the problem goes away. That is not what is happening here. The model is doing the exact job it was trained to do. The training just taught it a bad habit along the way.
The difference matters, because it changes what you write down. One team writes “the agent must not hallucinate”. Nobody can test that, and no model can meet it. Another team writes “95% of factual claims must trace back to a retrieved source”. A tester can run that on Monday. This primer is about getting to the second kind of wording. It builds on primer 1, which sets out the prediction loop in six steps.
Part 1 / Origin
Where it comes from, and why scale will not remove it
Published research splits the problem into two stages, and each stage needs a different fix. The first one is built in during pretraining. The second one happens afterwards, and we are the ones keeping it alive.
The origin: prediction over unlabelled text
Pretraining is next-word prediction across a huge pile of text, and none of that text is labelled. A folder of pet photos comes tagged “cat” or “dog”. This pile only ever shows fluent writing. Nothing in it is tagged “this claim was false”.
So the model has to infer what is valid, because it was never shown. For rare, arbitrary facts, no amount of extra data makes that inference reliable.
The persistence: grading that rewards the guess
Post-training tries to squeeze the guessing out. Then evaluation puts it straight back, because the leaderboards that matter score accuracy and nothing else.
When accuracy is the only score, a guess always beats saying nothing. Run that across thousands of benchmark questions and the model that guesses wins the table.
Why spelling errors disappeared and birthdays did not
Grammar and spelling follow steady patterns, so more data and a bigger model push those errors towards zero. Arbitrary facts have no pattern behind them. Nothing about the letters in a person’s name hints at their date of birth. A model trained to predict the next word cannot work one out if it never saw it often enough.
The paper uses a good comparison. Imagine training a classifier to label pet photos with the animal’s birthday. You can improve the algorithm as much as you like and the error rate will not fall, because the answer is not in the picture. The authors prove a version of the same point. The rate at which a generative model produces invalid statements can never sit much below twice the rate at which it would get a simple true-or-false call wrong.
Scales away
Errors with a pattern behind them
Spelling, grammar, formatting, syntax, common phrasing. The signal sits everywhere in the text and it is consistent, so scale takes the error rate down.
Does not scale away
Arbitrary, low-frequency facts
Dates, ID numbers, citations, prices, one-off credits, anything the model saw once or never. There is no pattern to generalise from, so it produces an answer that looks right rather than no answer at all.
Part 2 / Incentive
The scoreboard is teaching it to guess
Picture a multiple-choice exam where a blank scores zero and a guess has, say, a one in 365 chance of landing. There is no situation where leaving it blank is the smarter move. Almost every public benchmark sets up that same trade, and models are tuned to do well on public benchmarks.
The numbers are blunt. Here are two models from the same vendor on SimpleQA, a benchmark that tests factual recall, as reported in the GPT‑5 system card.
| Metric | gpt‑5‑thinking‑mini | o4‑mini |
|---|---|---|
| Abstention rate: said it did not know | 52% | 1% |
| Accuracy: the headline number | 22% | 24% |
| Error rate: confidently wrong | 26% | 75% |
The second model wins the published metric by two points. It also hands back a wrong answer three times as often. Ask it a question and three times in four you get something false, instead of one time in four. Ranked on accuracy, that counts as progress.
Where your organisation enters the story
You will build this same trap yourself without noticing. Say your acceptance criteria grade the agent on task completion, and “I could not determine that from the records available” gets logged as a failure. You have just copied the benchmark’s incentive into your own programme. Then you tune prompts, retrieval and routing until the guessing behaviour wins.
How you design evaluation inside your company shapes how honest your product is. It probably matters more than which vendor you pick.
Part 3 / Definition
A definition you can actually test against
Here is ours. We suggest you write it into your own documents:
Working definition
A hallucination is confident output that the evidence available to the system does not support. It comes out of the same machinery that produces the correct answers.
Two things follow from that wording, and both of them change how you write the test.
There is no made-up-answer module to switch off. The right answer and the invented one come out of the same loop, cost the same to produce, and read just as smoothly. How well a passage reads tells you almost nothing about whether it is true. That is why a review that skims the output and forms an impression will keep missing these.
A statement can be true and still unsupported. The model can hand you a fact that happens to be correct even though nothing in the retrieved evidence backs it up. For assurance work, unsupported is the thing you police. You cannot audit luck. A system that was right by chance today has given you nothing to rely on tomorrow.
Two analogies that hold up with technical audiences
- A curve fit with no error bars. Ask it for a value in a range where it never saw any data and it hands one back anyway, to the same number of decimal places as everywhere else. It has no way to say “that is outside my data”, because the fit keeps no record of where the data was.
- The exam candidate who never leaves a blank. They are not being dishonest. They are playing the marking scheme in front of them correctly. Change the marking scheme and the behaviour changes with it.
Part 4 / Controls
What to require instead
“The agent must not hallucinate” fails the first test of any requirement: nobody can tell you whether it has been met. Every row below is something a tester can run and a reviewer can sign off.
| Instead of | Write |
|---|---|
| “The agent must not hallucinate.” | Groundedness ≥ 0.95 on the curated evaluation set: every factual claim resolves to a span in a retrieved source. |
| “Answers must be accurate.” | Accuracy on the golden set reported alongside abstention rate and error rate. All three, always together. |
| “It should say when it is unsure.” | On the unanswerable subset, meaning cases with no supporting evidence in the corpus, the agent abstains or escalates in at least 95% of runs. |
| “Cite sources.” | Every citation resolves to a document that exists and contains the claim. Broken or non-supporting citations are counted and fail the gate. |
| “Get the numbers right.” | Arithmetic is recomputed deterministically outside the model and compared. A mismatch blocks the action rather than annotating it. |
And four habits behind those numbers
- Build an unanswerable set on purpose. A golden set made only of answerable questions cannot catch the failure this whole article is about. Ten to twenty per cent of your cases should have no supported answer, and declining is the right result on those.
- Make honest uncertainty a pass in the rubric. A confident wrong answer should score worse than “I could not determine that”. Where the stakes allow it, give the agent a clear “flag for review” path, and count using it as a success rather than as containment lost.
- Do not set temperature to zero and call it fixed. Getting the same answer every time is not the same as getting the right one. A repeatable wrong answer is still wrong, and the provider’s own infrastructure can break the repeatability anyway.
- Sort failures into named types. Unsupported claim, wrong entity, out-of-date record, misread evidence, right answer for the wrong reason. Counts per type tell you which control to build next. One lumped “hallucination rate” tells you nothing you can act on.
That change of framing is the whole point. Hallucination is not a defect you raise with a vendor and then wait on. It is a rate. You can measure it, budget for it, and hold it under a limit you agreed with the business, using controls you own.
Sources
- OpenAI and Georgia Tech. Why language models hallucinate, Kalai, Nachum, Vempala and Zhang (openai.com)
- OpenAI. GPT‑5 system card, for the SimpleQA abstention and error rates (openai.com)
- Anthropic. On the biology of a large language model, for the internal circuitry behind refusal and recall (transformer-circuits.pub)
- NotedML. Agentic AI Assurance, domain D1, for where groundedness sits in the wider framework
Next in the series
The model can explain itself. Whether the explanation is true is a separate question.
Primer 3 takes on the reasoning trace. Researchers have checked whether a model’s stated reasoning matches what actually drove its answer. Here is what they found, and what you can honestly claim in an audit once you know it.