Primer / 4 of 4
A data science mindset for agentic assurance
Trust the distribution, not the demo.
A lot of organisations are living through the same moment right now. A team demos an agent. It books the meeting, drafts the report, queries the database. It really is impressive. Then somebody asks the one question that matters. Can we trust it? And the room goes quiet, because nobody knows how to answer.
Here is our argument. If your organisation has ever shipped anything data-driven, whether that is analytics, a model, an A/B test or a serious spreadsheet, the instincts you need to answer that question are already in the building. Nobody has pointed them at agents yet. Underneath the smooth writing and the confident tool calls, these systems behave in ways a data scientist knows on sight. Distributions. Bias. Drift. Poor calibration. Scoring systems that quietly reward the wrong thing.
This is the last of four primers. The three before it covered how a model predicts, what hallucination really is, and whether a reasoning trace can be trusted. This one puts them together and turns them into practice.
Part 1 / The stance
Agents are stochastic systems under test, not minds under examination
Almost every mistake we see in agent assurance goes back to one wrong idea. People treat the agent as something that either knows or does not know, then test it the way they would test software that either works or does not.
Software is predictable. Give it the same input and it does the same thing, so a single run is a fair observation and a bug is something you reproduce and fix. An agent draws instead. Give it the same input and it does one of many things, pulled from a spread you have never measured. A single run is a single sample, and a failure is usually not a defect at all. It is part of that spread you had not looked at.
Think weather, not clockwork. Nobody asks a forecaster to get rid of rain. They ask how likely it is, how heavy, and how much warning they will get. Agent assurance has to move into that same way of talking, and your organisation already knows how to work there.
The reframe, in one line
You are not auditing a mind that might lie to you. You are estimating how a system that draws at random will behave, under conditions you set, with consequences you can limit.
Part 2 / Foundations
Four questions, and what the evidence says
The four primers in this series are not four explainers. They answer the four questions an assurance owner has to be able to answer about any agent you run.
Question 01
Why is it random?
Because writing text is a draw. The model works out a probability for every possible next token, then one gets picked, over and over. There is no fact store in that loop and nothing whose job is to check truth. Reading well and being right are two different things that usually, but not always, go together.
The consequence
Question 02
What can it actually see?
Less than the window promises. Controlled studies that changed only the input length found accuracy dropping on easy retrieval tasks, and dropping unevenly, with distractors and position mattering more as the context grew. Bigger windows moved the cliff edge rather than removing it.
The consequence
Question 03
How does it fail confidently?
Because nothing in training tells valid statements from invalid ones, and because accuracy-only scoring always makes a guess better than saying nothing. On SimpleQA, one model beat another by two points of accuracy while being wrong 75% of the time against 26%.
The consequence
Question 04
Can we believe its self-reports?
Not as evidence. When researchers planted hints that clearly changed the answer, the models mentioned them 25% and 39% of the time. That fell under 2% when what moved them was a scoring flaw they had learned to exploit.
The consequence
One more point sits under all four, and it is the one people skip most often. Assurance is not a model choice. Systems built from parts fail at the joins. A good model, a good retriever and a good set of tools can still add up to a bad run, and the monitor you bolt on top is usually another model with blind spots of its own. Picking a stronger model is not a control. It shifts the base rate and leaves every system-level failure exactly where it was.
Part 3 / The playbook
Six habits that turn the science into practice
Each foundation turns into something a team can start on Monday. These six are what we coach, in this order, and none of them need a research background.
Demo once, then measure distributions
Build a golden set of 100 to 500 realistic cases. Put in the awkward ones, the unclear ones, and the ones with no supported answer at all. Run every case several times. Report the spread, the slices and the failure rates. Never a single story, and never a single run.
Keep a frozen holdout set that nobody tunes against. Tweak against one evaluation set for long enough and you are teaching to the test without meaning to. You find out on the day the production numbers stop matching the dashboard.
Pick your statistic by risk tolerance
There are two ways to score repeated trials, and they answer different questions. pass@k asks whether the agent got it right at least once in k attempts. That is the measure you want when a person is in the loop and can try again. pass^k asks whether it got it right every single time. That is the measure you want for anything facing customers or running unattended.
The gap between the two is wider than most people expect.
| Per-trial success | All of 3 succeed | All of 5 succeed |
|---|---|---|
| 0.75 | 42% | 24% |
| 0.90 | 73% | 59% |
| 0.95 | 86% | 77% |
| 0.99 | 97% | 95% |
The same sums apply along a run of steps, not just across repeated attempts. An agent that takes twenty steps, each one 99% reliable on its own, finishes cleanly about 82% of the time. At 95% per step across fifty steps, it is under 10%.
| Per-step reliability | 5 steps | 20 steps | 50 steps |
|---|---|---|---|
| 0.99 | 95% | 82% | 61% |
| 0.95 | 77% | 36% | 8% |
| 0.90 | 59% | 12% | 0.5% |
Take these as a rough feel for the floor, not a forecast. Real agents break the assumption that steps are independent, and they break it both ways. Self-correction helps. Failures that cluster together hurt. What the table does justify is a budget for how many steps you allow, a stated reliability target for each step, and reporting whole-run success next to step-level numbers. Step-level dashboards flatter agents, and they flatter them most where the run is longest.
Grade abstention kindly
Write it into the rubric: a confident wrong answer scores worse than an honest “I am not sure”. Where the stakes allow it, give the agent a clear “flag for review” path and count taking it as a success rather than as a miss.
For most organisations this one change buys the most, because the default incentive points the other way. If your acceptance criteria mark “I could not determine that from the records available” as a failure, you will tune your way to a system that guesses. The whole time, the tuning will look like progress.
Put the verification in the harness, not in the prompt
Asking the model to be careful is not a control. Checking its work outside the model is one.
- Ask for structure, not prose. Require a schema instead of a free-form block of text, so a broken answer fails loudly rather than just reading well.
- Citations that resolve. Every reference has to point at a document that exists and actually contains the claim. Count the ones that do not, and fail the gate on them.
- Recompute the numbers yourself. Arithmetic, dates, entitlements and totals all get worked out again outside the model and compared. A mismatch blocks the action.
- Run your business rules on high-stakes fields. Take the rules you already have and apply them to the agent’s output before it turns into an action.
- Least privilege, plus approval gates. Keep tool permissions narrow by default, and make anything you cannot undo wait for a person. How much damage a failure can do is something you design, not something the model decides.
- Treat everything fetched as untrusted. Retrieved documents, tool responses and web content are all inputs an attacker might control. Instructions smuggled in through them, known as prompt injection, are still the classic way into an agent system.
Review failures like a scientist
Somebody senior should read several dozen real failures by hand every month. Not a summary of them. The actual transcripts. Totals tell you how often it goes wrong. Only the transcripts tell you how, and how is what you build a control against.
Keep a short list of failure types and count each one. Agree an error budget for each severity tier, with a named owner. And if you use an LLM as a judge or a monitor, check it against human labels before you trust the volume it gets through. A judge you have not checked is not a control, however many rows it produces.
Pin versions and watch for regressions
Your customers should never be the first to notice that the model changed underneath you. Track model and prompt versions explicitly. Keep the regression suite ready to run at any time. Roll upgrades out to a slice of traffic first. Re-run the golden set on every real change to any part of the system.
And hold the line on what the reasoning trace is for. Triage and debugging, always. Sign-off, never. Resist the urge to tidy traces up for an audience. The moment you score them on how they read, you are tuning appearance and throwing away the diagnostic value that made them worth keeping.
Part 4 / The default
Replace two superstitions with one discipline
The point of understanding the sampling loop, the attention budget and the incentives is not to turn product owners into ML engineers. It is to retire the two expensive habits that rush in whenever nobody in the room can answer the trust question.
The first is being dazzled. The demo was remarkable, the vendor sounds credible, ship it. The second is flat refusal. It makes things up, so we cannot use it for anything that matters. Both are guesses dressed up as judgement, and most organisations hold both at once in different departments.
The professional default is one you already practise elsewhere. Ask statistical questions. Ask for evidence. Limit the consequences. Trust what you can check yourself over what sounds certain.
The question to change
Stop asking “does it work?”
Start asking “how often, how badly, and how would we know before our customers do?”
Your organisation knows how to answer that question. It has answered it about forecasts, about credit models, about fraud rules and about every A/B test it has ever run. Agents are new. The discipline is not.
Sources
- OpenAI and Georgia Tech. Why language models hallucinate (openai.com)
- Anthropic. Reasoning models don’t always say what they think (anthropic.com)
- Multi-lab position paper. Chain-of-thought monitorability: a new and fragile opportunity for AI safety (arxiv.org)
- Anthropic. Demystifying evals for AI agents, for pass@k against pass^k (anthropic.com)
- Anthropic. Building effective agents, on workflows against agents and where combining parts goes wrong (anthropic.com)
- Chroma. Context Rot: how increasing input tokens impacts LLM performance (research.trychroma.com)
- OWASP. Top 10 for LLM Applications, for prompt injection and untrusted input (owasp.org)
Putting it to work
The habits tell you how to think. The framework tells you who signs.
Agentic AI Assurance covers the operating side. Eight assurance domains, four platform archetypes, three risk tiers and five lifecycle gates, plus a workbook of worksheets you fill in for each agent.