← All posts

There is a gap at the centre of enterprise AI, and IBM has been pointing at it for years: organisations deploy AI far faster than they govern it Source 1. The model gets shipped; the policy, the audit trail, and the cost ceiling arrive later — if at all.

That gap is usually filed as a compliance problem. It is also an efficiency problem, and that framing is the one most teams miss.

The ungoverned system

An ungoverned AI system has a recognisable shape: application code calls a model directly, with no layer in between. Which means:

• No policy. Any caller can invoke any model with any prompt, including ones that reach data classes they should never touch.
• No audit. When an answer is wrong, harmful, or expensive, there is no record of who asked what, or which model and version produced it.
• No cost ceiling. Token spend — or GPU-seconds, if you self-host — is unbounded. A retry loop or a runaway agent bills until someone notices the invoice.
• No attribution. You cannot say which team, feature, or agent drove the spend, so you cannot reduce it.

This is what "fast" looks like before governance: outputs arrive quickly, and you have no idea what they cost, whether they were allowed, or how to make them cheaper. That is efficiency theatre — the dashboard is green because nothing is measuring the parts that are red.

Governance as the efficiency layer

Reframe governance not as a brake but as the instrumentation that makes efficiency possible. You cannot optimise what you do not meter, and you cannot meter what flows through no chokepoint. So you add one.

The basic architecture is a single governed path that every model call passes through:

flowchart LR
  A[App / Agent] --> G[AI Gateway]
  G --> P{Policy Engine - OPA}
  P -- denied --> X[Reject and log]
  P -- allowed --> M[Model: hosted or API]
  M --> L[(Audit log)]
  M --> T[(Metering: tokens / GPU-seconds)]
  T --> R[Cost attribution per team and agent]

Five moving parts, each earning its place:

1. Gateway. One ingress for every model call. Without a chokepoint, none of the rest is enforceable — this is the decision everything else depends on.
2. Policy engine. Policy-as-code (Open Policy Agent is the common choice Source 2) decides allow or deny before the model runs: tool allowlists, data-class rules, per-caller budget caps. Rules live in version control, not in a wiki.
3. Audit log. Every request and response, with caller identity, model, and version — the record you need the day an answer causes a problem, and the accountability the NIST AI Risk Management Framework asks for Source 3.
4. Metering. Tokens for hosted APIs, GPU-seconds when you run your own. The unit matters: when the model is free but the GPU is the scarce resource, tokens are the wrong meter.
5. Cost attribution. Roll metering up per team, feature, and agent. This is where governance pays for itself.

Where the efficiency actually comes from

Once the path exists, the wins are mechanical, not hypothetical:

• Metering surfaces waste. Attribution turns "AI is expensive" into "this one agent is most of the spend, and half its calls are retries" — a sentence you can act on. You need the meter first; that is the whole point.
• Caps prevent the runaway. A budget rule in the policy engine stops the loop that would otherwise bill all night. Prevented cost is the cheapest cost.
• Policy enables autonomy. Counter-intuitively, the allowlist is what lets you give an agent more freedom: you can let it act because the blast radius is bounded, logged, and reversible.

Governance does not slow the system down. It is the difference between an AI system you can reason about and one that merely runs.

The takeaway

The IBM gap — deploy fast, govern later — is not a sequencing accident. Governance gets deferred because it is filed under risk, and risk is someone else's budget. File it under efficiency instead. The same gateway that enforces a policy is the one that meters the spend, and the same audit log that satisfies a reviewer is the one that tells you where your tokens went. Build the governed path first, and efficiency stops being a number on a slide and becomes something you can measure and improve.

Sources

1. What is AI governance?
2. policy-as-code for cloud-native systems.
3. AI Risk Management Framework AI RMF 1.0.

Built, then written

Tested on my own homelab before publishing — a four-architecture cluster (ARM · AMD ROCm · NVIDIA CUDA · Apple Silicon) running this blog, the RAG pipeline, and a sovereign research copilot. Built and tested before it's written — refined as I learn. See the platform →

Rafael Lopes

Production AI Engineer in Vancouver, BC. Brazilian. Builds and ships production AI on a self-hosted homelab — RAG pipelines, distributed LLM inference, web performance, and platform engineering.

Related posts