You’re in a board meeting. The CISO is presenting on AI risk. The CFO asks a simple question:

“When that finance agent we deployed last quarter accessed a customer payment record, can we tell who authorized it, what policy permitted it, and produce the full audit trail?”

The CISO looks at the head of the platform. The head of the platform looks at security. Nobody answers.

If you can picture that meeting happening at your company, you’re not alone. McKinsey found that only one-third of organizations have AI agent governance maturity at level 3 or higher. The other two-thirds are exactly the silence in that boardroom.

This post is the diagnostic framework that closes that gap. It’s part 2 of a five-part series on AI agent accountability, and if you only have time to read one post in the series, read this one. By the end you’ll have a five-question assessment to run with your team this week, and a maturity model to score where you stand today.

Not all governance equals AI agent accountability. Many enterprises believe they’re covered because they have network policies or an API gateway, but governance without accountability is a security theater: it might prevent some bad outcomes, but it can’t prove why good outcomes were permitted, trace what happened when something goes wrong, or satisfy an auditor asking for evidence.

True AI agent accountability requires five distinct capabilities working together. Miss any one and you have a gap that will surface during your next incident, audit, or regulatory review.

The five pillars are:

• Traceability: Every agent interaction produces an end-to-end record automatically.
• Authorization provenance: Every permitted action is traceable to a specific, auditable policy.
• Identity and ownership: Every agent has a verified identity and a clear human owner.
• Policy-based governance at scale: Declarative, attribute-based policies that don’t break at 100 agents.
• Human oversight and intervention: Humans can see, review, and override agent behavior in real time.

Each pillar comes with a question you can ask your team. Below, we’ll work through each one, and at the end, a 5-level maturity model and a 5-question assessment to score where you stand today.

Pillar 1: Traceability

“Can you trace what happened, end to end?”

When Agent A calls Agent B, which calls Tool C, which accesses Database D, can you reconstruct the entire chain? Not just that it happened, but when, how long each step took, and what the outcome was at each hop?

Traceability means every agent interaction produces a structured, correlated record automatically. This is distributed tracing applied to agent communication. Each hop in the chain is a span; the full trace tells the complete story of an interaction from trigger to outcome.

Without traceability, incident response is guesswork. You know something went wrong, but you can’t determine the chain of events that led there.

The test: Can your team pull up a single interaction and see the full path it took across every agent and tool in your network, with timestamps and outcomes at every hop?

Pillar 2: Authorization provenance

“Can you prove why it was permitted?”

Blocking unauthorized actions is table stakes. The harder (and more important) question is, can you prove why authorized actions were permitted?

Authorization provenance means every allowed interaction is traceable to a specific, auditable policy. Not just “Agent A was allowed to call Agent B,” but “Agent A was allowed to call Agent B because Policy X grants agents with capability Y access to agents with risk-level Z.”

This is the difference between a lock on the door and a sign-in sheet. The lock prevents unauthorized entry. The sign-in sheet proves who was authorized, when, and by what authority.

Without authorization provenance, your compliance team cannot demonstrate that access was intentional and governed, only that it wasn’t blocked. That distinction is the difference between passing an audit and failing one..

The test: For any agent-to-agent interaction in your network, can you identify the specific policy that permitted it and the attributes that triggered that policy?

Pillar 3: Identity and ownership

“Who owns this agent, and who is responsible when it acts?”

Every agent must have two things: a verified identity (it is who it claims to be) and a clear owner (a person accountable for its behavior).

Identity means the governance layer can verify that an agent is genuinely the agent it claims to be, and not a compromised workload masquerading as a legitimate one. This requires cryptographic identity verification, not just a name in a configuration file.

Ownership means that when an incident occurs, there is a specific person (not a team alias, not a Slack channel, not “the AI team”) who is accountable. Without clear ownership definitions, accountability diffuses across components, and diffused accountability is no accountability at all.

Agent registration should capture: who registered it, what team owns it, what it’s designed to do, and what permissions it’s been granted.

The test: Pick any agent in your network. Can you immediately identify it’s a verified identity, who registered it, which team owns it, and what permissions it has… all without asking around?

Pillar 4: Policy-based governance at scale

“Does your security model survive agent #101?”

With 10 agents, you can manage permissions by hand. You write explicit rules: “Agent A can call Agent B. Agent C can call Agent D.” You maintain a spreadsheet. It works.

With 100 agents, it doesn’t. With 1,000, it’s impossible. Every new agent requires updating every relevant policy. Permissions become a tangled web that nobody fully understands. New agents get deployed ungoverned because updating the allow-lists is too slow.

Scalable governance requires declarative, attribute-based policies. Instead of naming specific agents, policies reference agent attributes: capabilities, risk levels, teams, environments.

• “Low-risk agents can communicate with low-risk agents.”
• “Agents on the finance team can access finance MCP servers.”
• “Agents in production can only call production-grade tools.”

When a new agent registers with matching attributes, it’s governed from day one — automatically. No policy updates required. No spreadsheet to maintain. The governance scales with the agent network, not against it.

The test: When your team deploys a new agent next week, will it be governed by existing policies automatically, or will someone need to manually update an allow-list?

Pillar 5: Human oversight and intervention

“Can a human review, approve, or override?”

The EU AI Act (Article 14) requires effective human oversight of high-risk AI systems. But human oversight doesn’t mean a human approves every agent action, that would eliminate the value of agents entirely.

Effective human oversight means:

• Visibility: Humans can see what agents are doing, which agents are communicating, and what policies govern them.
• Review: Humans can examine agent interactions after the fact, with enough context to understand what happened and why.
• Intervention: Humans can modify policies, revoke agent access, or halt agent communication in real time when necessary.
• Dashboard, not log file: The oversight interface should be a visual dashboard with communication graphs and policy visualization, not a grep command on a log file.

The test: Right now, can someone on your team open a dashboard, see which agents are communicating with which, and modify the policies governing that communication — all without touching a terminal?

How to assess your AI agent accountability maturity

Run this five-question assessment with your platform lead, security lead, and one compliance representative in a 30-minute meeting. For each question, you have three possible answers: Yes (you’ve got it), Partial (you can answer for some agents but not all), or No (gap).

1. Pick the most recent agent-to-agent interaction in your environment. Can someone on the call pull up the full trace (every hop, timestamp, and outcome) in under five minutes? (Pillar 1)
2. For that same interaction, can you name the specific policy that permitted it and the agent attributes that triggered the match? (Pillar 2)
3. Pick one production agent at random. Can you produce (from a system, not a wiki) its verified identity, registered owner, team, and granted permissions? (Pillar 3)
4. Imagine your team deploys a brand-new agent tomorrow. Will your existing policies govern it automatically, or will someone need to update an allow-list? (Pillar 4)
5. Open whatever dashboard your team uses to view agent activity. Does it show communication graphs and policy state visually, or are you grep-ing a log file? (Pillar 5)

Count your answers. Five Yes = Level 4. Mostly Yes, occasional Partial = Level 3. Yes on identity but No on policy enforcement = Level 2. Inventory only, no identity verification = Level 1. Couldn’t run the assessment because you don’t know what agents exist = Level 0.

If you scored below Level 3, you’re in the McKinsey two-thirds. The good news: you now know exactly which pillar to fix first.

The Accountability Maturity Model

The five pillars map to a five-level progression. Use it to track where you are today and where you’re heading.

Most enterprises today are at Level 0 or Level 1. They lack verified identities, policy enforcement, and end-to-end auditability. The goal is Level 4, and the gap between where most organizations are and where they need to be is the AI agent accountability crisis this framework addresses.

Common questions

What is the most important pillar of AI agent accountability?
All five are required, but authorization provenance is the one most enterprises miss. Plenty of teams can block unauthorized actions; very few can show why an authorized action was permitted, traceable to a specific policy. Without provenance, you have security but not accountability.

How is AI agent accountability different from observability?
Observability tells you what happened. Accountability tells you what was permitted, by which policy, and on whose authority. Observability is a prerequisite, but it’s not enough on its own; your trace data needs to be tied to policy decisions and identity claims to count as accountability.

How does AI agent accountability relate to AI agent security?
They’re complementary, not interchangeable. AI agent security focuses on preventing compromise—stopping prompt injection, blocking unauthorized API access, eliminating shadow agents. Accountability focuses on proving what authorized agents did and why. You need both: security keeps the bad agents out, accountability keeps the good agents honest. The five pillars in this framework assume strong AI agent security is already in place.

Can I assess my AI agent governance maturity using these pillars?
Yes — that’s exactly what the assessment and maturity model above are for. Walk through each pillar’s “test” with your team. If you can’t answer cleanly on all five, you’re at Level 3 or below, regardless of what tooling you’ve deployed.

Do I need all five pillars on day one?
No, but you need a path to all five. A platform that delivers two pillars natively and forces you to bolt on the other three is an accountability gap waiting to surface. We cover what to look for in future articles of this series.

What is the difference between Level 3 and Level 4 in the maturity model?
Level 3 means unauthorized interactions are blocked, you have policy enforcement. Level 4 means you can also prove why every authorized interaction was permitted, with audit evidence tied to a specific policy and identity. Level 3 is security; Level 4 is accountability.

Key takeaways

• AI agent accountability rests on five pillars: traceability, authorization provenance, identity and ownership, policy at scale, and human oversight.
• Each pillar has a clear test you can run against your environment today.
• The five pillars map to a five-level Accountability Maturity Model — most enterprises are at Level 0 or 1.
• Run the 5-question assessment with your platform, security, and compliance leads to score where you stand.
• Missing any single pillar creates a gap that will surface during your next incident, audit, or regulatory review.

Get the strategic guide for accountable AI agents

We wrote a strategic guide for engineering and security leaders that goes deeper into each pillar, including detailed assessment questions, the full maturity model, and a practical roadmap to Level 4.

Accountable AI Agents: A Strategic Guide for AI & Security Leaders Governing Autonomous AI at Scale — no code, no product demos. Just the framework your leadership team needs.

Get the strategic guide for accountable AI agents →