Primer / 1 of 4

How a language model actually predicts

Next-token prediction in plain terms — and what it costs you to not know it.

Rob Cooper April 2026 9 min read

Read the assurance framework Talk it through with us

Ask an agent the same question twice and you can get two different answers. That is not a defect report. That is the design.

This is the first of four primers on the science underneath agent systems. Not enough to build a model — enough to have an instinct for where one will break, and to write tests that catch it. It starts here because everything else in the series follows from it: why context is an attention budget rather than a filing cabinet, why hallucination is a predictable consequence rather than a bug, and why a model’s stated reasoning is a working note rather than an audit log.

None of it requires maths. It requires being precise about one loop.

Part 1 / The loop

One token at a time, and nothing else

A large language model does one thing. Given everything in front of it, it produces a probability for every token that could come next, one token is drawn from that set of probabilities, and the loop runs again with that token added. Six steps, repeated until it stops.

Tokenise

Your input is cut into tokens — word fragments, punctuation, code symbols. Not words, and not characters.

Embed

Each token becomes a long list of numbers encoding how that token tends to get used. Tokens used alike sit near each other.

Attend

Layer after layer lets every token look at every other token in the context and work out which of them matter to it.

Predict

Out comes a score for every token in the vocabulary — often more than 100,000 candidates — turned into probabilities that sum to one.

Sample

One token is drawn from that distribution, usually weighted by probability. Likely tokens win often. Not always.

Repeat

The chosen token joins the context and the whole thing runs again. Every word you read was a separate draw.

Figure 1 — the generation loop. Steps 1 to 4 are deterministic given the input. Step 5 is where the randomness lives, and it is the only place it lives.

# the whole of it context = [your prompt] loop until done: # a probability for every possible next token probs = model(context) # draw ONE — the randomness is here token = sample(probs) context.append(token) return rendered_text

Figure 2 — the same loop as pseudocode. Nothing has been left out of it. There is no step where the model looks anything up, and no step whose job is to check whether what it just wrote is true.

Written formally, the model produces the probability of the next token given everything before it. That is the entire objective. It is worth sitting with what is absent from that description: there is no database being queried, no fact store being consulted, and no component anywhere in the pipeline whose responsibility is to separate true statements from plausible ones.

invoice18%

invoices9%

the7%

payment5%

…61%

Figure 3 — one step of prediction, illustrative. The tail is the point: the remaining probability is spread across roughly a hundred thousand other tokens, and on a long generation some of it gets drawn.

Part 2 / Sampling

Two consequences you have to build around

Fluent is not correct, and typical is not true

When a model states a fact, it is selecting language that is statistically typical of the text it was trained on. Most of the time typical and true coincide, which is exactly why the demo goes well. They are still two different properties, and nothing in the loop is checking the second one.

Fluency is therefore useless as a signal. A confident, well-structured, correctly formatted answer and a confident, well-structured, correctly formatted fabrication are produced by the same mechanism at the same cost. Any acceptance process that leans on how the output reads is measuring the wrong variable.

Run-to-run variance is the design, not a malfunction

Step 5 is a draw. Do it again and you can get a different token, and one different token early can send the rest of the generation somewhere else entirely. On easy tasks the answers stay close. On hard ones they diverge.

The published version of this is worth keeping in your pocket. Researchers asked a widely used chatbot for the title of one author’s own PhD dissertation and got three different answers across three attempts, none of them correct. Asked for his birthday, it gave three different dates, all wrong. Not three attempts at recall — three draws from a distribution that never contained the fact.

Attempt 01

A confident, well-formed answer. Wrong.

Attempt 02

A different confident, well-formed answer. Also wrong.

Attempt 03

A third. Wrong again, and no less certain than the first two.

Figure 4 — the same question, three times, as reported in the OpenAI hallucination paper. If your evaluation is one manual run, all three of these pass or all three fail depending on nothing but the draw.

The practical form of this: a single run tells you almost nothing. One run is one sample. You would not accept a single observation as evidence anywhere else in your business, and there is no reason to start here.

The two opposite mistakes

“It’s just autocomplete” undersells this badly. Anthropic’s interpretability work — effectively a microscope pointed at a production model — found real internal computation happening between the tokens: in one study the model planned the rhyming word at the end of a line before it began writing the line, and could be steered mid-plan.

So a narrow training objective produces surprisingly structured behaviour. Both errors here are expensive. Dismiss it as a parrot and you will under-build; assume it is checking itself against reality and you will not build the checks at all.

Part 3 / Context

The context window is an attention budget, not a filing cabinet

The context window is everything the model can see on this pass: your system instructions, the conversation so far, retrieved documents, the results that came back from tools. Current frontier windows run somewhere between 200,000 and a million tokens — loosely, 150,000 to half a million words and up. That sounds like a library, and the reasonable conclusion is to put everything in it.

The evidence says be more careful than that. Attention works by comparing token pairs, and pairs grow with the square of the length: a thousand tokens is about a million pairs, a hundred thousand tokens is about ten billion. The ability to use what is in the window thins out as the window fills.

What the controlled studies found

Length alone degrades performance. Chroma’s context-rot work held task difficulty constant and varied only input length across seventeen-plus models. Accuracy fell as inputs grew — on retrieval tasks trivial enough that difficulty could not be the explanation.
The degradation is not uniform. It depends on where the relevant passage sits, how much similar-but-irrelevant material surrounds it, and how the haystack is structured. Two models given the same long input do not fail the same way.
Distractors get worse with length. Content that resembles the answer without being the answer is mildly harmful in a short context and considerably more harmful in a long one.
Bigger windows moved the cliff, they did not remove it. Vendor documentation says this out loud: as tokens accumulate, recall from that context decreases.
Be honest about the disagreement. A 2026 preprint using tightly controlled synthetic probes found no length-driven decay out to 150,000 tokens on several frontier models. The robust claim is not “long context always fails”. It is that degradation shows up under realistic complexity, and that you cannot assume it away.

The mental model that survives contact

An open-book exam where you can see every page at a glance but can only actually read part of it — and where the book is wiped the moment the session ends.

Nothing persists between sessions. Anything durable has to live in external storage, and that storage brings its own failure surface: stale records, wrong retrievals, updates that never made it back in.

So context is a design variable

Context engineering is curation, not stuffing. What goes in, where it sits, and how much noise it arrives with are decisions — and decisions can be tested. Treat them with the seriousness you already give the retrieval algorithm, because they can cost you more than the retrieval algorithm does.

What lives in the window, and what each part can cost you
Occupant	Typical share	What goes wrong
System instructions	Small, constant	Diluted by everything appended after them; contradicted by retrieved text
Conversation history	Grows every turn	Grows without limit until something truncates it — usually silently
Retrieved documents	Largest and most variable	Distractors, near-duplicates, stale versions, wrong document entirely
Tool results	Spiky and unbounded	One verbose API response can crowd out the instructions
Prior reasoning	Large on reasoning models	Consumes budget that the actual evidence needed

Part 4 / Your tests

What changes on Monday

Five things follow directly from the loop and the window. None of them need a research team.

Run every case more than once. Fix a set of realistic inputs, run each of them n times, and report the spread as well as the mean. A case that passes four times in five is a different asset from a case that passes five times in five, and a single run cannot tell them apart.
Test at the length you will actually run at. An agent evaluated on a clean 2,000-token context and deployed against a 90,000-token one has not been evaluated. Include the noise, the near-duplicates and the long histories that production will hand it.
Vary position deliberately. Put the deciding fact at the start, in the middle and at the end of the context, and compare. If the answer depends on where you filed the evidence, you have found a real defect in the design and not a quirk of the model.
Stop treating fluency as a signal, in writing. Acceptance criteria should reference groundedness, citation resolution and rule adherence. If a reviewer’s note can be satisfied by rewording, the criterion was aesthetic.
Pin the model version and re-run the suite on every change. The loop is the same next month; the weights may not be. Version explicitly, keep the regression set, and canary the upgrade.

None of this makes the system deterministic. It is not going to be deterministic. It makes the system measured, which is the only honest basis for signing anything off.

Sources

Anthropic — Tracing the thoughts of a large language model, and On the biology of a large language model (anthropic.com, transformer-circuits.pub)
OpenAI and Georgia Tech — Why language models hallucinate, for the repeated-question anecdote (openai.com)
Chroma — Context Rot: how increasing input tokens impacts LLM performance (research.trychroma.com)
Anthropic — Effective context engineering for AI agents (anthropic.com)

Next in the series

If prediction is the whole loop, hallucination is not a bug.

The second primer takes the same mechanism and follows it to its most expensive consequence — why models make things up, why evaluation design keeps teaching them to, and what to police instead.

Read primer 2 The assurance framework