Primer / 1 of 4

How a language model actually predicts

Next-token prediction in plain terms, and what it costs you when you do not know it.

Rob Cooper April 2026 9 min read

Read the assurance framework Talk it through with us

Ask an agent the same question twice and you can get two different answers. Nobody broke anything. That is how it was built.

This is the first of four primers on the science underneath agent systems. It will not teach you to build a model. It will give you a feel for where one breaks, and help you write tests that catch it. We start here because the rest of the series rests on it. Why the context window is an attention budget and not a filing cabinet. Why models make things up. Why a model’s stated reasoning is a working note and not an audit log.

You will not need any maths. You will need to be exact about one loop.

Part 1 / The loop

One token at a time, and nothing else

A large language model does one thing. It looks at everything in front of it and works out a probability for every token that could come next. One token gets drawn from those probabilities. Then the loop runs again with that token added on. Six steps, over and over, until it stops.

Tokenise

Your input is cut into tokens: word fragments, punctuation, code symbols. Not whole words, and not single letters.

Embed

Each token turns into a long list of numbers that captures how the token tends to get used. Tokens used alike end up near each other.

Attend

Layer after layer lets every token look at every other token in the context and work out which of them matter to it.

Predict

Out comes a score for every token in the vocabulary, often more than 100,000 candidates. Those scores become probabilities that add up to one.

Sample

One token is drawn from that distribution, usually weighted by probability. Likely tokens win often. Not always.

Repeat

The chosen token joins the context and the whole thing runs again. Every word you read was a separate draw.

Figure 1. The generation loop. Given the same input, steps 1 to 4 always give the same result. Step 5 is where the randomness lives, and it is the only place it lives.

# the whole of it context = [your prompt] loop until done: # a probability for every possible next token probs = model(context) # draw ONE. the randomness is here token = sample(probs) context.append(token) return rendered_text

Figure 2. The same loop written as pseudocode. Nothing has been left out. There is no step where the model looks anything up, and no step whose job is to check whether what it just wrote is true.

Put formally, the model works out the probability of the next token given everything before it. That is the whole goal it was trained on. Now look at what is missing from that description. No database gets queried. No fact store gets checked. Nowhere in the pipeline is there a part whose job is to tell a true statement apart from one that merely sounds right.

invoice18%

invoices9%

the7%

payment5%

…61%

Figure 3. One step of prediction, for illustration. The tail is the point. That leftover probability is spread across roughly a hundred thousand other tokens, and over a long piece of writing some of it gets drawn.

Part 2 / Sampling

Two consequences you have to build around

Fluent is not correct, and typical is not true

When a model states a fact, it is picking wording that is typical of the text it was trained on. Most of the time typical and true line up, which is exactly why the demo goes so well. They are still two different things, and nothing in the loop is checking the second one.

So how well the answer reads tells you nothing. A confident, well-structured, correctly formatted answer and a confident, well-structured, correctly formatted invention come out of the same machinery for the same cost. Any sign-off process that leans on how the output reads is measuring the wrong thing.

Run-to-run variance is the design, not a malfunction

Step 5 is a draw. Run it again and you can get a different token. One different token early on can send the rest of the answer somewhere else entirely. On easy tasks the answers stay close together. On hard ones they spread apart.

There is a published example worth keeping in your pocket. Researchers asked a widely used chatbot for the title of one author’s own PhD thesis. Three attempts gave three different answers, and none of them was right. Asked for his birthday, it gave three different dates, all wrong. Those were not three attempts to remember something. They were three draws from a set of probabilities that never held the fact in the first place.

Attempt 01

A confident, well-formed answer. Wrong.

Attempt 02

A different confident, well-formed answer. Also wrong.

Attempt 03

A third. Wrong again, and no less certain than the first two.

Figure 4. The same question, three times, as reported in the OpenAI hallucination paper. If your evaluation is a single manual run, whether you see a pass or a fail comes down to nothing but the draw.

Here is the practical version. A single run tells you almost nothing. One run is one sample. You would not take a single observation as evidence anywhere else in your business, and there is no reason to start here.

The two opposite mistakes

“It’s just autocomplete” sells this far too short. Anthropic’s interpretability work points something like a microscope at a working model, and it found real computation going on between the tokens. In one study the model picked the rhyming word for the end of a line before it started writing the line, and researchers could steer it part way through.

So a narrow training goal can produce surprisingly organised behaviour. Both mistakes cost you. Write it off as a parrot and you will under-build. Assume it is checking itself against reality and you will not build the checks at all.

Part 3 / Context

The context window is an attention budget, not a filing cabinet

The context window is everything the model can see on this pass: your system instructions, the conversation so far, retrieved documents, whatever came back from tools. Today’s leading windows hold somewhere between 200,000 and a million tokens, which is roughly 150,000 to half a million words. That sounds like a library, so the obvious move is to put everything in it.

The evidence says go carefully. Attention works by comparing tokens in pairs, and the number of pairs grows with the square of the length. A thousand tokens is about a million pairs. A hundred thousand tokens is about ten billion. As the window fills up, the model’s ability to actually use what is in there gets thinner.

What the controlled studies found

Length on its own hurts performance. Chroma’s context-rot work kept the task difficulty the same and changed only the input length, across more than seventeen models. Accuracy dropped as inputs grew, on retrieval tasks so easy that difficulty could not explain it.
The drop is uneven. It depends on where the relevant passage sits, how much similar but useless material sits around it, and how the pile is organised. Two models given the same long input will not fail in the same way.
Distractors hurt more as length grows. Content that looks like the answer without being the answer does mild damage in a short context and much more in a long one.
Bigger windows moved the cliff edge rather than removing it. Vendor documentation says so plainly: as tokens pile up, recall from that context goes down.
Be honest about the disagreement. A 2026 preprint used tightly controlled synthetic tests and found no length-driven decay out to 150,000 tokens on several leading models. So the safe claim is not that long context always fails. It is that the decay turns up once the task gets realistically messy, and you cannot just assume it away.

The mental model that survives contact

Think of an open-book exam where you can see every page at a glance but can only really read part of it. And the book gets wiped the moment the session ends.

Nothing carries over between sessions. Anything that has to last must live in outside storage, and that storage brings its own ways to fail: out-of-date records, wrong retrievals, updates that never made it back in.

So context is a design variable

Context engineering means choosing carefully, not cramming. What goes in, where it sits, and how much noise comes with it are all decisions, and decisions can be tested. Give them the same attention you already give the retrieval algorithm. They can cost you more than it does.

What lives in the window, and what each part can cost you
Occupant	Typical share	What goes wrong
System instructions	Small, constant	Watered down by everything added after them, and contradicted by retrieved text
Conversation history	Grows every turn	Grows with no limit until something cuts it off, usually without telling you
Retrieved documents	Largest and most variable	Distractors, near-duplicates, out-of-date versions, the wrong document entirely
Tool results	Spiky and unbounded	One long-winded API response can crowd out the instructions
Prior reasoning	Large on reasoning models	Eats budget that the actual evidence needed

Part 4 / Your tests

What changes on Monday

Five things follow straight from the loop and the window. None of them need a research team.

Run every case more than once. Fix a set of realistic inputs, run each one n times, and report the spread alongside the average. A case that passes four times in five is a very different thing from one that passes five times in five, and a single run cannot tell them apart.
Test at the length you will really run at. An agent tested on a clean 2,000-token context and then shipped against a 90,000-token one has not been tested. Put in the noise, the near-duplicates and the long histories that production will hand it.
Move the key fact around on purpose. Put it at the start, in the middle and at the end of the context, then compare the results. If the answer depends on where you filed the evidence, you have found a real design fault rather than a quirk of the model.
Write down that fluency does not count. Acceptance criteria should name groundedness, citation resolution and rule adherence. If a reviewer’s note can be cleared by rewording, the criterion was about style.
Pin the model version and re-run the suite on every change. The loop will be the same next month. The weights may not be. Set the version explicitly, keep the regression set, and roll upgrades out to a small slice first.

None of this makes the system give the same answer every time. It never will. What it does is make the system measured, and that is the only honest ground to sign anything off from.

Sources

Anthropic. Tracing the thoughts of a large language model, and On the biology of a large language model (anthropic.com, transformer-circuits.pub)
OpenAI and Georgia Tech. Why language models hallucinate, for the repeated-question example (openai.com)
Chroma. Context Rot: how increasing input tokens impacts LLM performance (research.trychroma.com)
Anthropic. Effective context engineering for AI agents (anthropic.com)

Next in the series

If prediction is the whole loop, hallucination is not a bug.

Primer 2 takes the same machinery and follows it to the most expensive place it leads. Why models make things up, how our own evaluation keeps teaching them to, and what to police instead.

Read primer 2 The assurance framework