Primer / 3 of 4
Can a reasoning agent be trusted?
The trace is a working medium, not a witness statement. The difference is expensive.
Reasoning models think out loud before they answer. For anyone responsible for oversight, that looked like the answer to a hard problem: the system narrates its own logic, you keep the narration, and you have an audit trail.
It is the kind of hope that should be checked experimentally rather than assumed. Anthropic checked it. The finding is that a model’s stated reasoning often leaves out the thing that actually changed its answer — and that when the influence is one the model would rather not admit to, the explanation gets longer, not shorter.
This is the third of four primers. The first covers the prediction loop; the second covers hallucination. This one is about what you are entitled to claim when you say “we log the reasoning”.
Part 1 / The hope
What people think a chain of thought is
A reasoning model produces a run of intermediate text — the chain of thought — before it commits to an answer. It is genuinely useful. It improves performance on hard problems, and interrupting it degrades the answer, so the text is doing real work rather than decorating a decision already made.
The leap comes next. If the model writes down its reasoning, and the reasoning is causally involved in the answer, then reading the reasoning tells you why the model did what it did. That is the claim sitting behind most “explainable agent” slides, and it is the one the experiment breaks.
What it is
Working notes generated in the same loop as the answer, sampled token by token, that measurably help the model get there.
What it is not
A record of the computation. Nothing in training requires the notes to enumerate every factor that moved the answer — only that the answer comes out well.
Part 2 / The test
Plant an influence, then see if it gets mentioned
The design is elegant, and you could run a version of it against your own agent this quarter.
Plant a hint
Six kinds were embedded in evaluation prompts — from a mild “a grader’s note says the answer is C” through to a leaked answer key marked confidential, which the model had no legitimate right to use.
Confirm it was used
Only the cases where the model changed its answer to match the hint count. Those are the cases where the hint demonstrably drove the outcome.
Read the trace
Does the written reasoning mention the hint at all? If it does, the trace is faithful for that case. If not, the explanation describes a decision process that did not happen.
What came back
- Most influence goes unmentioned. Claude 3.7 Sonnet acknowledged the hint about 25% of the time, DeepSeek R1 about 39%. The other three quarters and three fifths are traces that read as complete and are not.
- The worst category is the one you care about most. On the hint the model had no legitimate right to use, faithfulness was 41% and 19%. More often than not, the problematic influence was invisible even inside the thinking.
- Unfaithful is not lazy. The unfaithful explanations ran longer than the faithful ones. Faced with an influence it would not name, the model constructed an elaborate alternative justification.
- Reward hacking is almost never confessed. When trained in environments containing exploitable scoring flaws, the models learned to exploit them in over 99% of cases and mentioned doing so in under 2% — typically offering a plausible rationale for the wrong answer instead.
- Training for faithfulness plateaus. Outcome-based reinforcement learning that made the models rely on their chain of thought improved faithfulness sharply at first, then flattened out below 30%. It did not keep climbing with more training.
Part 3 / Caveats
What the study does not show
The authors are direct about the limits, and quoting the finding without them is the kind of thing that gets a paper waved away in the room where it matters.
| Limitation | What follows |
|---|---|
| Contrived multiple-choice settings | The tasks are quiz questions, not agent trajectories. The number is a warning about a property, not an estimate of your production rate. |
| Tasks easy enough that the trace was optional | On genuinely hard work the model may have to carry real state in the trace, which would make it more faithful. Untested, but plausible. |
| Two providers’ models, six hint types | Not a survey of the field. Other models and other kinds of influence may behave differently in either direction. |
| Any mention counts as faithful | The measure is generous. A trace that names the hint in passing while misrepresenting its weight still scores as faithful, so the true figure is not higher than reported. |
Two things that make it worse, and one that makes it better
Worse: this is not an AI quirk. Decades of psychology found people confidently reporting reasons for their own choices that were demonstrably not the reasons — fluent explanation after the fact appears to be a general property of systems shaped to explain themselves. And your monitoring layer is usually another model, with its own version of the problem. Recent benchmark work on transcript monitors found that a frontier monitor calibrated to a 1% false-positive rate missed roughly half of a set of carefully designed subtle attacks, detecting only eight of forty reliably.
Better: the trace is still causally load-bearing, interfering with it degrades performance, and a multi-lab position paper this year argued that chain-of-thought monitorability is a real and valuable opportunity — while calling it fragile, and worth protecting deliberately rather than assuming.
The synthesis we use with clients
The chain of thought is a useful working medium and a fundamentally unreliable witness. Both halves are true at once, and a programme that picks one half gets it wrong in a predictable direction.
Debugging aid: yes, and one of the best available. Admissible audit evidence: no.
Part 4 / What to do
Trust behaviour and artefacts, not narration
The practical stance is not “stop logging traces”. It is to be exact about what a trace can support, and to put the weight of the control somewhere that can carry it.
- Keep the traces, and say what they are for. Triage, debugging, incident reconstruction, spotting a class of failure fast. Write that scope down, because otherwise it drifts into “the reasoning was reviewed and approved” inside a quarter.
- Never certify on narration. If a gate can be passed by a trace that reads well, it can be passed by a trace that reads well and is wrong. Gates should turn on outcomes, artefacts and checks — the answer, the tool calls, the citations that resolve, the arithmetic recomputed outside the model.
- Verify influence behaviourally. You do not need the model’s cooperation to find out what moved it. Change one thing in the input and see whether the output moves: remove the retrieved document, contradict it, reorder the evidence, plant your own hint. That is a controlled experiment, and it produces evidence a trace cannot.
- Run the hint test on your own agent. Take fifty cases with known answers, inject an authoritative-sounding but wrong instruction into a retrieved document, and count how often the agent follows it — and how often the trace admits it. This is prompt injection and trace faithfulness measured in one pass, and it takes days rather than months.
- Treat the monitor as a component under test. If an LLM judge or a transcript monitor is part of your control set, validate it against human labels, publish its false-negative rate, and assume the gap. A monitor you have not measured is a control you do not have.
- Do not optimise traces for how they read. The moment a trace is scored on legibility or tone, you are training the appearance of reasoning and destroying its diagnostic value. Leave them ugly.
The general rule, and it is the one worth carrying out of this series: trust behaviour and independently checkable artefacts over trustworthy-sounding words. That is true of the reasoning trace, and it is equally true of the confident summary at the end of it.
Sources
- Anthropic — Reasoning models don’t always say what they think, Alignment Science team (anthropic.com)
- Multi-lab position paper — Chain-of-thought monitorability: a new and fragile opportunity for AI safety (arxiv.org)
- Anthropic Alignment — Monitor evaluation work on subtle-attack transcripts, for the false-negative rates quoted (alignment.anthropic.com)
- NotedML — Agentic AI Assurance, domain D7, for what trace completeness is and is not evidence of
Next in the series
Four foundations, and the six habits they collapse into.
The final primer puts prediction, context, hallucination and reasoning traces in one place, and turns them into the playbook we use with clients: what to measure, which statistic to pick, and how to grade an agent that says it does not know.