Primer / 4 of 4
A data science mindset for agentic assurance
Trust the distribution, not the demo.
There is a moment a lot of organisations are living through right now. A team demonstrates an agent. It books the meeting, drafts the report, queries the database. It is genuinely impressive. Somebody asks the only question that matters — can we trust it? — and the room goes quiet, because nobody knows how to answer.
Our argument is that if your organisation has ever shipped anything data-driven — analytics, a model, an A/B test, a serious spreadsheet — the instincts needed to answer that question are already in the building. They just have not been pointed at agents yet. Underneath the fluent prose and the confident tool calls, these systems have properties a data scientist recognises on sight: distributions, bias, drift, miscalibration, and evaluation regimes that quietly reward the wrong thing.
This is the last of four primers. The three before it covered how a model predicts, what hallucination actually is, and whether a reasoning trace can be trusted. This one puts them together and turns them into practice.
Part 1 / The stance
Agents are stochastic systems under test, not minds under examination
Almost every mistake we see in agent assurance traces back to one category error: treating the agent as a thing that either knows or does not know, and testing it the way you would test software that either works or does not.
Software is deterministic. Given the same input it does the same thing, so one run is a valid observation and a bug is a defect to reproduce and fix. An agent samples. Given the same input it does a thing, drawn from a distribution you have not characterised. One run is one sample, and a failure is usually not a defect at all — it is a tail you had not measured.
The useful analogy is weather rather than clockwork. Nobody demands that a forecaster eliminate rain. They ask how likely, how heavy, and how much warning. That is the register agentic assurance has to move into, and it is one your organisation can already operate in.
The reframe, in one line
You are not auditing a mind that might lie to you. You are estimating the behaviour of a stochastic system, under conditions you control, with consequences you can constrain.
Part 2 / Foundations
Four questions, and what the evidence says
The four primers in this series are not four explainers. They are answers to the four questions an assurance owner has to be able to answer about any agent in the estate.
Question 01
Why is it random?
Because generation is a draw. The model produces a probability for every possible next token and one is sampled, repeatedly. There is no fact store in that loop and nothing whose job is to check truth. Fluent and correct are separate properties that usually, but not always, coincide.
The consequence
Question 02
What can it actually see?
Less than the window advertises. Controlled studies varying only input length found accuracy falling on trivial retrieval tasks, unevenly, with distractors and position mattering more as the context grows. Bigger windows moved the cliff rather than removing it.
The consequence
Question 03
How does it fail confidently?
Because nothing in training separates valid from invalid statements, and because accuracy-only scoring makes a guess strictly better than an abstention. On SimpleQA, one model beat another by two points of accuracy while being wrong 75% of the time against 26%.
The consequence
Question 04
Can we believe its self-reports?
Not as evidence. When researchers planted hints that demonstrably changed the answer, the models acknowledged them 25% and 39% of the time — and under 2% when the influence was an exploit they had learned to use.
The consequence
One more thing sits underneath all four, and it is the one most often skipped: assurance is not a model choice. Composed systems fail at the level of the composition — a good model, a good retriever and a good tool set can still produce a bad trajectory, and the monitor you put on top is usually another model with blind spots of its own. Choosing a stronger model is not a control. It changes the base rate and leaves the system-level failure modes exactly where they were.
Part 3 / The playbook
Six habits that turn the science into practice
Each foundation collapses into something a team can do on Monday. These six are what we coach, in this order, and none of them require a research background.
Demo once, then measure distributions
Build a golden set: 100 to 500 realistic cases, including the awkward ones, the ambiguous ones and the ones with no supported answer. Run every case multiple times. Report spreads, slices and failure rates — never a single anecdote and never a single run.
Keep a frozen holdout that nobody tunes against. Repeated iteration on one evaluation set quietly becomes teaching to the test, and the day you notice is the day the production numbers disagree with the dashboard.
Pick your statistic by risk tolerance
There are two ways to score repeated trials and they answer different questions. pass@k asks whether the agent succeeded at least once in k attempts — the right measure when a human is in the loop and can retry. pass^k asks whether it succeeded every time — the right measure for anything customer-facing or unattended.
The gap between them is larger than intuition suggests.
| Per-trial success | All of 3 succeed | All of 5 succeed |
|---|---|---|
| 0.75 | 42% | 24% |
| 0.90 | 73% | 59% |
| 0.95 | 86% | 77% |
| 0.99 | 97% | 95% |
The same arithmetic runs along the trajectory rather than across attempts. An agent that takes twenty steps, each independently 99% reliable, completes cleanly about 82% of the time. At 95% a step across fifty steps it is under 10%.
| Per-step reliability | 5 steps | 20 steps | 50 steps |
|---|---|---|---|
| 0.99 | 95% | 82% | 61% |
| 0.95 | 77% | 36% | 8% |
| 0.90 | 59% | 12% | 0.5% |
Treat these as floor intuition rather than a forecast. Real agents violate independence in both directions — self-correction helps, correlated failure modes hurt. What the table justifies is a step budget, an explicit reliability target per step, and reporting end-to-end trajectory success alongside step-level metrics. Step-level dashboards flatter agents, and they flatter them most exactly where the trajectory is longest.
Grade abstention kindly
Make confidently wrong score worse than honestly uncertain, in the rubric, in writing. Where the stakes allow it, give the agent an explicit “flag for review” path and count taking it as a success rather than as containment lost.
This is the single highest-leverage thing most organisations can change, because the incentive is currently pointing the other way by default. If your acceptance criteria mark “I could not determine that from the records available” as a failure, you will tune your way to a system that guesses — and the tuning will look like progress the whole time.
Put the verification in the harness, not in the prompt
Asking the model to be careful is not a control. Checking its work outside the model is.
- Structure over prose. Require a schema rather than a free-form blob, so a malformed answer fails loudly instead of reading plausibly.
- Citations that resolve. Every reference has to point at a document that exists and contains the claim. Count the ones that do not; fail the gate on them.
- Deterministic recomputation. Arithmetic, dates, entitlements and totals get recalculated outside the model and compared. A mismatch blocks the action.
- Business-rule validators on high-stakes fields. The rules you already have, applied to the agent’s output before it becomes an action.
- Least privilege and approval gates. Tool scopes narrow by default; anything irreversible waits for a human. The blast radius is a design parameter, not a property of the model.
- Everything fetched is untrusted. Retrieved documents, tool responses and web content are inputs an attacker may control. Instructions smuggled through them — prompt injection — remain the classic route into an agentic system.
Review failures like a scientist
Somebody senior should read several dozen real failures by hand every month. Not a summary of them — the transcripts. Aggregate metrics tell you the rate; only the transcripts tell you the mechanism, and the mechanism is what you build a control against.
Keep a simple failure taxonomy and count by category. Agree error budgets per severity tier, with a named owner for each. And if you deploy an LLM as a judge or a monitor, validate it against human labels before you trust its throughput — an unvalidated judge is a control you do not have, however many rows it produces.
Pin versions and watch for regressions
An upstream model change should never be the first thing your customers notice. Track model and prompt versions explicitly, keep the regression suite runnable on demand, canary upgrades against a slice of traffic, and re-run the golden set on every material change to any part of the system.
And hold the line on what the reasoning trace is for. Triage and debugging, always. Certification, never. Resist the urge to tidy traces for presentation — the moment they are scored on how they read, you are optimising appearance and losing the diagnostic value that made them worth keeping.
Part 4 / The default
Replace two superstitions with one discipline
The point of understanding any of this — the sampling loop, the attention budget, the incentive gradients — is not to turn product owners into ML engineers. It is to retire the two expensive postures that fill the vacuum when nobody in the room can answer the trust question.
The first is dazzled capitulation: the demo was remarkable, the vendor is credible, ship it. The second is reflexive distrust: it makes things up, we cannot use it for anything that matters. Both are guesses dressed as judgement, and an organisation usually holds both at once, in different departments.
The professional default is the one you already practise elsewhere. Ask statistical questions. Demand evidence. Constrain consequences. Trust what you can independently verify over what sounds certain.
The question to change
Stop asking “does it work?”
Start asking “how often, how badly, and how would we know before our customers do?”
That is a question your organisation knows how to answer. It has answered it about forecasts, about credit models, about fraud rules and about every A/B test it has ever run. Agents are new. The discipline is not.
Sources
- OpenAI and Georgia Tech — Why language models hallucinate (openai.com)
- Anthropic — Reasoning models don’t always say what they think (anthropic.com)
- Multi-lab position paper — Chain-of-thought monitorability: a new and fragile opportunity for AI safety (arxiv.org)
- Anthropic — Demystifying evals for AI agents, for pass@k against pass^k (anthropic.com)
- Anthropic — Building effective agents, on workflows against agents and where composition fails (anthropic.com)
- Chroma — Context Rot: how increasing input tokens impacts LLM performance (research.trychroma.com)
- OWASP — Top 10 for LLM Applications, for prompt injection and untrusted input (owasp.org)
Putting it to work
The habits tell you how to think. The framework tells you who signs.
Agentic AI Assurance sets out the operating side of this: eight assurance domains, four platform archetypes, three risk tiers and five lifecycle gates, with a workbook of worksheets you complete per agent.