Primer / 3 of 4
Can a reasoning agent be trusted?
The reasoning trace is a working note, not a witness statement. Mistaking one for the other gets expensive.
Reasoning models think out loud before they answer. If you are responsible for oversight, that looks like a gift. The system talks through its own logic, you keep the transcript, and you have an audit trail.
That is a hope worth testing rather than assuming, and Anthropic tested it. They found that a model’s stated reasoning often skips the very thing that changed its answer. Worse, when the influence is one the model would rather not own up to, the explanation gets longer rather than shorter.
This is the third of four primers. Primer 1 covers the prediction loop and primer 2 covers hallucination. This one is about what you can honestly claim when you say “we log the reasoning”.
Part 1 / The hope
What people think a chain of thought is
Before it settles on an answer, a reasoning model writes a stretch of working text. People call it the chain of thought. It genuinely helps. Models do better on hard problems with it, and cutting it short makes the answers worse, so the text is doing real work rather than dressing up a decision already made.
Then comes the leap. If the model writes its reasoning down, and that reasoning helps produce the answer, then reading it must tell you why the model did what it did. That claim sits behind most “explainable agent” slides, and it is the one the experiment breaks.
What it is
Working notes, made in the same loop as the answer, drawn token by token, that measurably help the model get there.
What it is not
A record of what the model computed. Training never asks the notes to list every factor that moved the answer. It only asks for a good answer.
Part 2 / The test
Plant an influence, then see if it gets mentioned
The design is simple, and you could run a version of it against your own agent this quarter.
Plant a hint
Six kinds were slipped into the test prompts. They ran from a mild “a grader’s note says the answer is C” up to a leaked answer key marked confidential, which the model had no business using at all.
Confirm it was used
Only count the cases where the model changed its answer to match the hint. In those cases the hint clearly drove the result.
Read the trace
Does the written reasoning mention the hint at all? If it does, the trace is faithful for that case. If it does not, the explanation describes a decision that never happened.
What came back
- Most of the influence never gets mentioned. Claude 3.7 Sonnet admitted the hint about 25% of the time, DeepSeek R1 about 39%. The remaining traces read as though they were complete. They were not.
- The worst case is the one you care about most. On the hint the model had no right to use, faithfulness came in at 41% and 19%. More often than not, the influence you would most want flagged was invisible even inside the thinking.
- An unfaithful trace is not a lazy one. The unfaithful explanations ran longer than the faithful ones. When the model would not name what moved it, it built a detailed story about something else instead.
- Reward hacking almost never gets confessed. Trained in settings with exploitable scoring flaws, the models learned to exploit them in over 99% of cases. They mentioned doing so in under 2%. Usually they offered a reasonable-sounding case for the wrong answer instead.
- Training for faithfulness stalls. Outcome-based reinforcement learning pushed the models to lean on their chain of thought. Faithfulness improved sharply at first, then flattened out below 30%. More training did not keep it climbing.
Part 3 / Caveats
What the study does not show
The authors are open about the limits. Quote the finding without them and someone in the room will wave the whole paper away.
| Limitation | What follows |
|---|---|
| Artificial multiple-choice settings | The tasks are quiz questions, not real agent runs. Read the number as a warning about a behaviour, not as an estimate of your own rate. |
| Tasks easy enough that the trace was optional | On genuinely hard work the model may have to keep real working state in the trace, which would make it more faithful. Nobody has tested that yet, but it is plausible. |
| Two providers, six hint types | This is not a survey of the field. Other models and other kinds of influence could go either way. |
| Any mention counts as faithful | The bar is low. A trace that names the hint in passing while playing down how much it mattered still scores as faithful, so the real figure is no better than reported. |
Two things that make it worse, and one that makes it better
First the bad news. This is not an AI quirk. Decades of psychology experiments have caught people confidently giving reasons for their own choices that were provably not the reasons. Smooth explanation after the fact seems to be what you get from any system built to explain itself. On top of that, the thing watching your agent is usually another model with the same weakness. Recent benchmark work on transcript monitors found that a leading monitor, tuned to a 1% false-alarm rate, missed roughly half of a set of carefully built subtle attacks. It reliably caught only eight out of forty.
Now the good news. The trace still does real work, and interfering with it makes performance worse. A position paper from several labs this year argued that watching the chain of thought is a real and valuable opportunity. The same paper called it fragile, and said it is worth protecting on purpose rather than taking for granted.
How we put it to clients
The chain of thought is a useful working document and an unreliable witness. Both of those are true at the same time, and a programme that holds on to only one of them goes wrong in a way you can predict.
As a debugging aid it is one of the best things you have. As audit evidence it does not stand up.
Part 4 / What to do
Trust behaviour and artefacts, not narration
None of this means stop logging traces. It means being exact about what a trace can back up, and putting the weight of the control somewhere that can hold it.
- Keep the traces, and write down what they are for. Triage, debugging, working out what happened in an incident, spotting a type of failure quickly. Put that scope in writing, because within a quarter it will otherwise turn into “the reasoning was reviewed and approved”.
- Never sign off on the narration. If a well-written trace can get through a gate, then a well-written and wrong trace can get through the same gate. Gates should turn on results, artefacts and checks: the answer, the tool calls, the citations that resolve, the arithmetic recomputed outside the model.
- Test what moved it by changing the input. You do not need the model’s cooperation for this. Change one thing and see whether the output moves. Take out the retrieved document. Contradict it. Reorder the evidence. Plant your own hint. That is a controlled experiment, and it gives you evidence a trace never can.
- Run the hint test on your own agent. Take fifty cases with known answers. Slip an official-sounding but wrong instruction into a retrieved document. Count how often the agent follows it, and how often the trace admits it did. That measures prompt injection and trace faithfulness in one pass, and it takes days rather than months.
- Treat the monitor as a component under test. If an LLM judge or a transcript monitor is one of your controls, check it against human labels, publish how often it misses, and plan around that gap. A monitor you have not measured is not a control.
- Do not tune traces for how they read. The moment you score a trace on how clear or polished it is, you are training the look of reasoning and throwing away its diagnostic value. Leave them ugly.
Here is the rule worth carrying out of this whole series. Trust what the system does, and artefacts you can check yourself, over words that sound trustworthy. That applies to the reasoning trace, and just as much to the confident summary at the end of it.
Sources
- Anthropic. Reasoning models don’t always say what they think, Alignment Science team (anthropic.com)
- Multi-lab position paper. Chain-of-thought monitorability: a new and fragile opportunity for AI safety (arxiv.org)
- Anthropic Alignment. Monitor evaluation work on subtle-attack transcripts, for the miss rates quoted (alignment.anthropic.com)
- NotedML. Agentic AI Assurance, domain D7, for what a complete trace is and is not evidence of
Next in the series
Four foundations, and the six habits they add up to.
The last primer puts prediction, context, hallucination and reasoning traces in one place and turns them into the playbook we use with clients. What to measure, which statistic to pick, and how to grade an agent that says it does not know.