Framework / Version 1.0
Agentic AI Assurance
Testing, ownership and governance for systems that act.
A chatbot answers. An agent acts. That single difference drives everything below.
An agentic AI system pairs a large language model with the ability to plan multi-step work, call tools and APIs, read and write enterprise data, hold memory across steps, and decide its own path to a goal. Ask a chatbot about a refund policy and it retrieves text. Ask an agent to process the refund and it reads the order, checks entitlement, issues the credit, and logs the case. The blast radius of a wrong answer just became the blast radius of a wrong action.
This is the framework we use to decide what to test, who tests it, how hard, and when — and it is deliberately platform-neutral. It works the same whether you are writing agent code against a model API or configuring an agent in a SaaS console; what changes is who does the doing, not what has to be true before you ship. The whole thing is set out here. The workbook PDF adds the worksheets you copy per agent.
Part 1 / Primer
What an agent is, and why it breaks your test playbook
Anatomy of an agent
Every agent, regardless of platform, is built from the same six parts. Testing responsibility maps onto these parts — which is why the anatomy is worth agreeing on before anything else.
01
Model
The LLM that reasons, plans and generates.
Failure surface
Hallucination; silent vendor updates; capability gaps.
02
Instructions
Prompts, topics, guidelines and persona.
Failure surface
Ambiguity; conflicting rules; injection overriding them.
03
Tools & actions
APIs, flows and functions it can invoke.
Failure surface
Wrong tool; wrong parameters; excessive permissions.
04
Knowledge & grounding
Retrieval indexes, records and documents used as evidence.
Failure surface
Stale or wrong sources; retrieval failures; poisoned content.
05
Memory & state
Context carried across steps and sessions.
Failure surface
Context loss; state corruption; cross-user leakage.
06
Orchestration
The plan–act–retry loop that sequences and terminates.
Failure surface
Loops; dead ends; role drift; cascading multi-agent failures.
Why traditional testing fails here
Your existing QA regime almost certainly assumes determinism. Same input, same output, assert equality, done. Agents void that assumption four ways.
- Non-deterministic outputs. The same prompt can produce different, equally valid responses. Pass/fail assertions give way to scored evaluation against criteria, often run multiple times per scenario.
- Trajectory matters, not just the answer. An agent can reach the right outcome via a dangerous path — calling a payment API it never needed, for example. You must evaluate the full execution trace: every reasoning step, tool call and retrieval, not only the final message.
- The system changes underneath you. Model providers update models on their own schedule, retrieval sources change daily, and business rules evolve. A test suite that passed at go-live proves nothing three months later. Regression evaluation must run continuously, not at release.
- Adversaries are part of the input space. Users will attempt prompt injection, data exfiltration and guardrail bypass, deliberately or by accident. Security testing is a first-class discipline, not an afterthought.
Independent research keeps confirming the stakes. Gartner projects that by 2028 around 40% of enterprise AI failures will trace back to inadequate evaluation and monitoring rather than model capability. Surveys through 2026 consistently show most organisations piloting agents while fewer than one in five reach production scale, with quality and trust cited as the leading blockers. The gap is not model intelligence. It is assurance discipline.
The core mindset shift
Treat evaluation as a product deliverable, not a project phase. An agent you cannot measure is an agent you cannot ship, and an agent you stop measuring is an agent you no longer control.
The ownership problem
This is the failure pattern we see most often. The product team owns the use case. The data team owns the pipelines. The ML or AI team owns the model and prompts. Security owns “compliance”. Brand owns tone. Everyone owns a piece; nobody owns the whole. Each team assumes another team is validating decision quality, and the agent ships with systematic gaps that surface as incidents.
Three structural causes drive it:
- Agents cut across organisational seams. A single customer conversation touches brand voice, business rules, security boundaries, data governance and platform operations in one exchange. No existing team spans all of that.
- Platforms blur the build/buy line. A Salesforce admin can configure an Agentforce agent in days without engineering. Who then performs security testing? The admin doesn’t know how; the security team doesn’t know it exists.
- Vendor guardrails create false comfort. Built-in trust layers and content filters cover generic harms. They do not know your refund policy, your regulatory obligations or your brand voice. Someone must test what the vendor cannot.
Everything that follows is a direct answer to that pattern: a shared model of what must be assured (domains), who assures it (roles and RACI), how much assurance each agent needs (risk tiers), and when it happens (lifecycle gates) — adjusted for how the agent is built (platform archetypes).
Part 2 / The framework
Four moving parts, one question each
Used together, the four parts answer the question every adopting organisation asks: what do we test, who tests it, how hard, and when?
| Element | Question it answers | Where |
|---|---|---|
| Platform archetypes | How is this agent built, and what does that imply for control and responsibility? | Part 2 |
| Assurance domains | What are the eight things that must be tested for every agent, regardless of platform? | Part 2 |
| Roles and RACI | Who is accountable, responsible, consulted and informed for each domain? | Part 3 |
| Risk tiers and gates | How much rigour does this agent need, and at which lifecycle points is it enforced? | Part 4 |
The principle underneath all four
You can delegate control implementation to a vendor; you can never delegate accountability. When Agentforce’s trust layer filters a toxic response, Salesforce ran the control — your organisation remains accountable for the customer outcome. Every schema here is built on that distinction.
Platform archetypes: the build spectrum
Agent solutions sit on a spectrum from full custom engineering to configured SaaS. Where an agent sits determines how much of the assurance stack you must build yourself versus verify the vendor has built. We use four archetypes.
A1 / Bespoke engineered
Custom code on model APIs; agent frameworks
Built by — engineering teams
You hold everything: model, orchestration, guardrails, tracing, evals.
A2 / Platform-assisted
Azure AI Foundry; Databricks Mosaic AI
Built by — engineering + data/AI
Platform supplies harnesses and primitives; you configure and extend.
A3 / Low-code maker
Copilot Studio; Power Platform agents
Built by — makers, fusion teams
Managed safety layer; you control topics, connectors, DLP, environments.
A4 / SaaS-native
Salesforce Agentforce; ServiceNow AI Agents
Built by — admins, consultants
Vendor owns engine and base guardrails; you configure topics, actions, scope.
The archetype trap
Teams pick an archetype for speed, then apply the testing regime of a different archetype. A bespoke-grade eval harness is overkill for a Copilot Studio FAQ agent; an admin’s preview-window spot check is negligent for an Agentforce agent issuing refunds. Match the regime to the archetype and the risk tier, not to habit.
The eight assurance domains
Every agent must be assured across eight domains. The domains are constant; the depth of testing (risk tier) and the division of labour (archetype) vary. This is the schema to socialise first, because it gives every team a shared vocabulary for the word “tested”.
D1
Task quality & capability
Goal completion, correctness, groundedness, trajectory.
D2
Business rules & policy
Your rules, thresholds, escalations. No vendor can test this.
D3
Brand, tone & experience
Voice, empathy, refusal style, handover to humans.
D4
Safety & responsible AI
Harm categories, bias and fairness, AI disclosure.
D5
Security & adversarial
Injection, jailbreaks, tool misuse — red teamed, multi-attempt.
D6
Data protection & privacy
Permission inheritance, PII masking, residency, retention.
D7
Traceability & audit
Reconstruct every step. If you can’t trace it, you can’t govern it.
D8
Operations & drift
Latency, cost, drift after updates, regression, rollback.
D1
Task quality and capability
Does the agent achieve the user’s goal, correctly and grounded in real data? Core measures: goal completion rate, answer correctness, groundedness (no hallucination beyond retrieved evidence), retrieval relevance, and trajectory quality — did it take a sensible path, select the right tools, pass the right parameters. Tested via curated evaluation datasets, LLM-as-judge scoring calibrated against human review, and perturbation testing: rephrase the input, and the behaviour should hold.
D2
Business rules and policy adherence
Does the agent follow your rules: eligibility criteria, approval thresholds, escalation triggers, regulated wording, jurisdictional constraints? Vendors cannot test this domain for you, because they do not know your rules. Tested via scenario suites derived from the business rule catalogue, including boundary cases — a refund at exactly the 30-day limit — and forbidden-action probes.
D3
Brand, tone and experience
Does the agent sound like you and behave like your best staff member? Covers voice, formality, empathy in sensitive moments, handling of complaints, refusal style, and conversational quality: containment without frustration, sensible handover to humans. Tested via rubric-scored transcript review by brand and CX owners, tone-calibrated LLM judges, and pilot user feedback.
D4
Safety and responsible AI
Does the agent avoid harm: toxic content, bias and unfair treatment across customer groups, inappropriate advice (medical, legal, financial), and manipulation? Tested via harm-category test suites, fairness testing across demographic slices, and disclosure checks — users know they are talking to AI. Anchors to your Responsible AI policy and, in Australia, the Voluntary AI Safety Standard guardrails.
D5
Security and adversarial resilience
Does the agent resist attack? Covers prompt injection (direct, and indirect via retrieved content), jailbreaks, tool misuse, privilege escalation, data exfiltration through outputs, and supply-chain exposure via connectors and MCP tools. Tested via structured red teaming — manual plus automated attack generation — OWASP LLM Top 10 coverage, and penetration testing of the surrounding integration. NIST’s agent red-teaming findings are blunt: single-shot testing understates risk. Run multi-attempt campaigns and refresh attack libraries continuously.
D6
Data protection and privacy
Does the agent respect data boundaries? Covers permission inheritance — the agent must not see or reveal more than the requesting user could — PII handling and masking, data residency, retention of transcripts and traces, and consent. Tested via permission-boundary probes across user personas, PII leakage scans on outputs and logs, and DLP policy verification.
D7
Traceability, observability and audit
Can you reconstruct, after the fact, what the agent did and why? Covers end-to-end tracing of every reasoning step, tool call, retrieval and decision; immutable audit logs; lineage from data source to answer; and evidence retention that satisfies your regulators. This domain is the precondition for incident response and for every other domain’s ongoing measurement. If you cannot trace it, you cannot govern it.
D8
Operations, drift and lifecycle
Does the agent stay healthy in production? Covers latency, cost per interaction, error and retry rates, drift detection (behavioural change after model or data updates), continuous regression evaluation on production traces, capacity, and version management with rollback. Tested via production monitoring with quality scorers running on live traffic, alert thresholds, and scheduled re-certification.
Part 3 / Ownership
Roles, RACI and shared responsibility
Eight roles cover the assurance work. Small organisations combine roles in one person; the roles still need naming, because an unnamed role is an unowned domain.
| Role | Typical home | Assurance mandate |
|---|---|---|
| Use Case Owner | Product or business unit | Accountable for the agent’s business outcome and its responsible use. Defines success criteria, approves scope and autonomy level, owns the risk acceptance. |
| Brand & Experience Lead | Marketing / CX | Owns voice, tone, persona and conversational quality standards. Signs off D3 and co-defines D1 success criteria. |
| Agent Engineering / Build Team | Engineering, or makers and admins for A3–A4 | Builds and configures the agent. Responsible for implementing controls, building eval suites, and fixing findings across all domains. |
| AI Platform Team | Data & AI / IT platform | Owns the shared platform: environments, model access, eval and tracing infrastructure, guardrail primitives, connector governance, agent registry. |
| Security (CISO org) | Information security | Accountable for D5 and co-accountable for D6. Runs or commissions red teaming, sets security standards per archetype, approves connectors and tool permissions. |
| Risk, Legal & Compliance | Risk / legal | Accountable for regulatory adherence, records obligations, AI-specific regulation (EU AI Act where applicable), and the risk tiering methodology. |
| AI Governance Council | Cross-functional: exec sponsor, platform, security, risk, business | Sets policy, owns the framework itself, arbitrates tier disputes, reviews Tier 1 agents at gates, maintains the agent inventory. |
| Operations / Service Management | IT operations / SRE | Runs D8: monitoring, alerting, incident response, drift review, decommissioning. |
The RACI schema
Below is the default allocation across the eight domains. Copy it per agent and adjust: the archetype shifts who is Responsible, never who is Accountable. One A per row, always.
| Domain | Use Case Owner | Brand / CX | Build Team | Platform | Security | Risk & Legal | Council | Ops |
|---|---|---|---|---|---|---|---|---|
| D1 Task quality | A | C | R | C | I | I | I | I |
| D2 Business rules | A | C | R | I | I | C | I | I |
| D3 Brand & tone | C | A | R | I | I | I | I | I |
| D4 Safety / RAI | A | C | R | C | C | C | C | I |
| D5 Security | C | I | R | C | A | C | I | C |
| D6 Data & privacy | C | I | R | R | C | A | I | I |
| D7 Traceability | C | I | R | A | C | C | I | R |
| D8 Operations | C | I | C | C | I | I | I | A/R |
Three allocations deserve explanation. The Use Case Owner is accountable for what the agent does (D1, D2, D4) because autonomy never transfers accountability away from the business. Security is accountable for resistance to attack regardless of who built the agent. The Platform Team is accountable for traceability because audit infrastructure must be shared, consistent and impossible for individual build teams to opt out of.
Shared responsibility by archetype
The RACI names who is accountable. This schema shows how the responsible work splits between your teams and the vendor as you move across the build spectrum.
The detail table below unpacks that shift domain by domain. Read each cell as “who does the doing”.
| Domain | A1 bespoke | A2 Foundry / Databricks | A3 Copilot Studio | A4 Agentforce |
|---|---|---|---|---|
| D1 Task quality | You build the full eval harness and datasets | You build datasets; platform supplies judges, scorers and tracing (Foundry evaluators, MLflow judges) | You script test cases; platform provides the test surface and analytics | You author Testing Centre test sets; vendor supplies the harness; you still define pass criteria |
| D2 Business rules | Fully yours | Fully yours — custom judges encode the rules | Fully yours — topic and flow testing | Fully yours — topic, instruction and action testing. No vendor knows your rules |
| D3 Brand & tone | Fully yours | Yours; tunable judges help automate rubric scoring | Yours | Yours; prompt builder configures tone, you verify it |
| D4 Safety / RAI | You select and validate all filters and run bias testing | Platform content safety plus your configuration and bias testing | Managed safety layer plus your verification on your scenarios | Einstein Trust Layer baseline plus your verification on your scenarios |
| D5 Security | You engineer and red-team all defences | Platform primitives (prompt shields, red teaming agents) plus your campaigns on your attack surface | Platform DLP and connector controls plus your injection testing via channels | Vendor platform security plus your testing of actions, permissions and injection paths |
| D6 Data & privacy | You engineer permission and masking layers | Platform governance (Unity Catalog, Purview) plus your permission-boundary probes | Environment strategy, DLP policy plus your persona-based probes | Sharing model and masking config plus your persona-based probes |
| D7 Traceability | You build tracing end to end (OpenTelemetry and similar) | Platform tracing (Foundry Observability, MLflow Tracing) plus your retention and evidence config | Platform audit logs plus your export and retention pipeline | Vendor audit trail plus your export, retention and gap analysis |
| D8 Operations | You build all monitoring and drift detection | Platform monitoring plus your thresholds, scorers and runbooks | Platform analytics plus your KPIs and review cadence | Vendor dashboards plus your KPIs, review cadence and version watch |
The pattern to internalise
Moving right across the spectrum, the vendor absorbs more of the how — harnesses, guardrails, logs — while you retain all of the what: your rules, your data boundaries, your pass criteria, your evidence obligations. D2 never moves. Verification never moves. Only implementation moves.
Part 4 / Rigour and rhythm
Risk tiering and lifecycle gates
Applying one control set to every agent fails in both directions. Heavy controls on a low-risk internal helper kill adoption; light controls on a customer-facing agent with financial actions invite incidents. Tier every agent at intake, and let the tier set the depth of testing at every gate.
Score the agent on five dimensions. The highest single dimension sets the tier — risk does not average out. For Tier 1, regulatory exposure includes privacy law, financial services obligations and EU AI Act high-risk categories where they apply.
Tier 3 — low
Internal, read-only, human executes
Streamlined checklist · owner self-attestation · automated guardrail checks · annual review.
Tier 2 — medium
Broad internal, acts with approval
All domains at standard depth · security testing to Security’s playbook · six-monthly review.
Tier 1 — high
Customer-facing, autonomous, sensitive data or regulated
Full eight-domain assurance · independent red team · Council sign-off at G2 and G3 · quarterly re-certification.
- Audience
- internal experts → customer or public-facing
- Autonomy
- drafts only → executes autonomously
- Data sensitivity
- non-sensitive → PII, financial or regulated
- Action impact
- read-only → financial or irreversible
- Regulatory exposure
- none → explicit regulation applies
Every agent, even Tier 3, enters the agent inventory. An unregistered agent is the single biggest audit failure we see — which is the whole reason we built MCPeer, our registry for the MCP servers and skills an agent can reach.
The five lifecycle gates
Gates convert the framework from a document into an operating rhythm. Each gate has an owner, entry criteria and evidence. No evidence, no pass. The Governance Council joins G2 and G4 for Tier 1 agents.
G0
Intake & tiering
Register, tier, confirm archetype, name the RACI.
G1
Design review
Permissions, guardrails, tracing plan and eval plan designed in.
G2
Pre-deployment assurance
Full domain test plan: evals, rules, tone, red team, probes.
G3
Controlled release
Canary cohort, transcript review, rollback rehearsed.
G4
Operate & re-certify
Monitor drift, regression on live traces, re-certify on trigger.
Gate 2, in full
Gate 2 is where most programmes are weakest, so here is its checklist in full. Scale depth by tier; skip nothing at Tier 1.
| Domain | Minimum evidence at Gate 2 | Sign-off |
|---|---|---|
| D1 Task quality | Eval run report on the curated dataset: goal completion, correctness and groundedness above agreed thresholds; trajectory review of failures | Use Case Owner |
| D2 Business rules | Scenario suite mapped to the rule catalogue, 100% of critical rules covered, boundary cases included, all passing or waived in writing | Use Case Owner + Risk |
| D3 Brand & tone | Rubric-scored transcript sample reviewed by Brand; refusal and complaint scenarios reviewed; persona consistency confirmed | Brand & Experience Lead |
| D4 Safety / RAI | Harm-category suite results; bias testing across relevant slices; AI disclosure verified in every channel | Use Case Owner + Risk |
| D5 Security | Red team report (multi-attempt campaigns, direct and indirect injection, tool misuse); findings remediated or accepted by Security; connector and permission review | Security |
| D6 Data & privacy | Permission-boundary probe results across user personas; PII leakage scan of outputs and logs; retention and residency config verified | Risk + Security |
| D7 Traceability | End-to-end trace of test conversations reproduced from logs; audit export tested; evidence retention meets the applicable schedule | Platform Team |
| D8 Operations | Monitoring dashboards live; alert thresholds set; cost model agreed; rollback rehearsed; support runbook published | Operations |
Part 5 / Tooling and standards
What the platform gives you, and what it doesn’t
Use this map to decide what you configure versus what you procure or build. “Native” means the capability ships with the platform — you still own configuring it, defining pass criteria, and covering the gaps. Verify current product capability at design time; this market moves quarterly.
| Capability | Foundry (A2) | Databricks (A2) | Copilot Studio (A3) | Agentforce (A4) |
|---|---|---|---|---|
| Evaluation harness | Foundry evaluators, Agents Playground evals, CI/CD integration | MLflow evaluation, AI judges, tunable and custom judges, Judge Builder | Test surface in Studio; scripted test cases; Power CAT tooling | Agentforce Testing Centre: batch test sets, AI-generated cases |
| Guardrails | Content safety filters, prompt shields, groundedness checks | AI Gateway guardrails, Unity Catalog policy enforcement | Managed safety layer, DLP policies, connector guardrails | Einstein Trust Layer: masking, toxicity detection, zero-retention options |
| Tracing / audit | Foundry Observability, Azure Monitor integration | MLflow Tracing on every step, lineage via Unity Catalog | Audit logs via Purview / Power Platform admin | Event logs, audit trail, utterance analysis |
| Red teaming | AI Red Teaming Agent (automated adversarial testing) | Bring your own — PyRIT, garak, promptfoo integrate well | Bring your own via channel testing | Bring your own against sandbox; vendor runs platform-level testing |
| Production monitoring | Continuous evaluation on live traffic, alerting | Lakehouse monitoring, scorers on production traces | Analytics dashboards, conversation KPIs | Agentforce analytics, utterance analysis dashboards |
| Human feedback loop | Playground plus your review workflow | Review App for SME labelling | Your review workflow on transcripts | In-console feedback capture plus your review workflow |
For A1 bespoke builds, assemble the equivalent stack yourself: an eval framework (promptfoo, DeepEval, Braintrust or similar), OpenTelemetry-based tracing (Langfuse, Arize Phoenix), guardrail libraries, and red-team tooling (PyRIT, garak). Expect two to four weeks of engineering to stand up credible evaluation infrastructure before the first production agent — and treat that as a platform investment shared across use cases, not a per-agent cost.
Standards to anchor to
Do not invent your control language from scratch. Anchor the framework to external standards so audits, procurement and regulators can read it.
| Standard | What it gives you | Use it for |
|---|---|---|
| NIST AI RMF (+ CAISI agent red-teaming guidance) | Risk management vocabulary — Govern, Map, Measure, Manage — and emerging agent-specific security testing findings | Overall risk framing; D5 test design (multi-attempt campaigns, refreshed attack libraries) |
| ISO/IEC 42001 | Certifiable AI management system: lifecycle controls, impact assessment, continual improvement | Programme structure; the certification procurement teams increasingly ask for |
| OWASP Top 10 for LLM Applications | Concrete vulnerability taxonomy: prompt injection, insecure output handling, excessive agency | D5 coverage checklist for every red team scope |
| EU AI Act | Legal obligations by risk category for systems touching the EU market | Tiering inputs; transparency and documentation requirements for in-scope agents |
| AU Voluntary AI Safety Standard | Ten guardrails for Australian organisations: accountability, testing, transparency, contestability, records | Board-level narrative and the D4/D7 baseline for Australian deployments |
| Privacy Act 1988 (Cth) / APPs | Australian privacy obligations on collection, use, disclosure and security of personal information | D6 requirements for any agent touching personal information in Australia |
Part 6 / Worksheets
The three things you copy per agent
These are deliberately short; the discipline is in filling them honestly, not in their length. The workbook PDF has them as blank forms.
6.1 / Gate 0
Use case intake and tiering
- Agent name and one-line purpose
- Use Case Owner — a named individual, not a team
- Platform archetype (A1–A4) and platform
- Audience: internal / partner / customer
- Autonomy level: drafts / approved actions / autonomous
- Data touched: classifications, PII yes or no
- Actions the agent can execute — list every one
- Regulatory context
- Proposed risk tier and rationale
- RACI confirmed for all eight domains (attached)
- Registered in the agent inventory, with an ID
6.2 / Gate 2 input
Test plan schema
One row per domain. A domain with no rows in your plan is a domain you have decided not to test; make that decision visible and signed.
| Element | What to record |
|---|---|
| Domain | D1–D8 |
| Test approach | Eval dataset / scenario suite / rubric review / red team campaign / probe set |
| Dataset or scenario source | Where cases come from: rule catalogue, historical tickets, synthetic generation, attack library |
| Coverage target | For example: 100% of critical business rules; all OWASP LLM Top 10 categories; all user personas |
| Pass criteria | Quantified thresholds — groundedness ≥ 0.9; zero critical injection successes |
| Method | Automated (judges, scorers), human review, or hybrid; number of runs per scenario |
| Responsible / Accountable | Per the agent’s RACI |
| Evidence artefact | Report or export retained as gate evidence, and where it lives |
6.3
Core metric catalogue
| Metric | Domain | Definition | Typical Tier 1 target |
|---|---|---|---|
| Goal completion rate | D1 | Share of scenarios where the user’s end-to-end objective was achieved | ≥ 90% on curated set |
| Groundedness | D1 | Share of factual claims supported by retrieved evidence | ≥ 95% |
| Trajectory accuracy | D1 | Share of runs taking an acceptable tool and step path | ≥ 90% |
| Rule adherence | D2 | Share of rule scenarios passed, criticals weighted | 100% critical; ≥ 98% overall |
| Tone rubric score | D3 | Mean brand-rubric score on sampled transcripts | ≥ 4.0 / 5.0 |
| Containment with satisfaction | D3 | Share resolved without human handover, gated by a CSAT floor | Set per channel; never containment alone |
| Harmful output rate | D4 | Share of harm-suite probes producing policy-violating output | 0% critical categories |
| Injection success rate | D5 | Share of red-team attempts achieving their objective (multi-attempt) | 0% critical; downward trend |
| Permission breach count | D6 | Probes returning data beyond the requester’s entitlement | Zero, always |
| Trace completeness | D7 | Share of production interactions fully reconstructable from logs | 100% |
| Drift alert rate | D8 | Significant behavioural deltas per release or model update | All investigated within SLA |
| Cost per resolved interaction | D8 | Fully loaded cost divided by successful resolutions | Set per business case |
Part 7 / Standing this up
The first 90 days
- Days 1–30 — establish the foundations. Convene the Governance Council, adopt the eight-domain vocabulary, build the agent inventory and register every existing agent (expect to find more than you knew about), and tier them. Publish the RACI schema and get each function to accept its accountabilities in writing.
- Days 31–60 — instrument and pilot. Stand up shared evaluation and tracing infrastructure per archetype in use. Run one existing Tier 1 or Tier 2 agent through Gate 2 retrospectively as the pilot. The retrofit will surface gaps; fixing them calibrates the checklist.
- Days 61–90 — operationalise. Wire Gates 0–4 into your delivery process: intake forms, pipeline checks, release approvals. Train makers and admins on their responsibilities for A3 and A4 agents; they are your largest unmanaged testing population. Schedule re-certification. Report the first assurance dashboard to the executive: agents by tier, gate status, open findings, incidents.
The risk to watch
Shadow agents. Low-code platforms let any licensed user create an agent this afternoon. Governance that only covers the agents you commissioned covers a shrinking fraction of the agents you run. Inventory, environment strategy and maker training are not bureaucracy; they are the perimeter.
References and further reading
- NIST — AI Risk Management Framework and CAISI agent security guidance (nist.gov)
- ISO/IEC 42001:2023 — AI management systems (iso.org)
- OWASP — Top 10 for LLM Applications (owasp.org)
- Australian Government — Voluntary AI Safety Standard (industry.gov.au)
- Microsoft — Azure AI Foundry Observability; agentic AI maturity model (learn.microsoft.com)
- Databricks — MLflow 3 GenAI evaluation and monitoring documentation (docs.databricks.com)
- Salesforce — Agentforce Testing Centre and Einstein Trust Layer documentation (help.salesforce.com)
- IMDA Singapore — Model AI Governance Framework for Agentic AI (imda.gov.sg)
The workbook
Parts 1 and 2 you read once. Parts 3 to 6 you fill in.
The PDF carries the same framework plus the worksheets as blank forms: the intake and tiering sheet, the test plan schema, and the metric catalogue, ready to copy into your own programme and complete per agent.