Why the next phase of enterprise AI will be won through runtime control, not model selection
Three years ago, the central question in enterprise AI was which model to use. Teams ran bake-offs, debated benchmarks, and built roadmaps around individual vendors.
Model selection still matters, particularly for specialized and high-risk applications. But for many enterprise workflows, it is no longer the primary constraint. Models are improving rapidly, costs are declining, and modern architectures increasingly allow organizations to route between models without rebuilding the entire application.
The model problem is increasingly being solved by well-capitalized technology companies moving at a pace and focus no enterprise can match internally.
Which leaves the harder question: can the enterprise control the systems built around those models?
Across over 300 interviews and enterprise implementations, I have seen nearly every troubled AI deployment begin to break down outside the model.
No one can say how many AI systems are running in the company. A retrieval pipeline is pointed at a SharePoint site whose permissions have not been reviewed since 2019. A developer connects an agent to production using a service account with far more authority than the task requires. A model powering a customer-facing workflow is deprecated or replaced, and no one re-validates the system. The policy intended to prevent all of this lives in a PDF the runtime has never read.
These are not model problems. They are control problems.
They are also the same class of problem that every prior infrastructure transition eventually had to confront. Enterprise networks needed network management. Cloud required identity, orchestration, and observability. Data platforms required lineage, quality controls, and access governance. The technology arrived first. The control layer arrived second, usually after something expensive went wrong.
AI is at that point now. The difference is speed. Agents are moving into production faster than prior enterprise technologies, and unlike a virtual machine or a data warehouse, they can take action on their own.
What is missing is a formal discipline for controlling AI as an operational system rather than treating governance as a set of policy commitments.
I define this emerging discipline as AI Control Theory.
AI Control Theory is the study and practice of how organizations maintain effective control over AI systems while those systems perceive information, make decisions, and take action.
It borrows its operating logic from control systems engineering. A control system compares the actual state of a process against a desired state. When the two diverge, the system intervenes.
Enterprise AI requires the same structure:
In this formulation, policy establishes the reference state. Runtime telemetry serves as the sensor. The control plane functions as the controller. Enforcement mechanisms are the actuators. Continuous evaluation measures the error between intended and actual behavior and closes the feedback loop.
Governance establishes what should happen. Control determines what is permitted to happen, intervenes when it should not, and produces evidence of what actually occurred.
That distinction sounds academic until an agent moves money.
A functioning AI control system performs four functions continuously: Sense, Decide, Act, and Verify.
Control begins with visibility, and most organizations do not have it.
Models are embedded in applications. Employees use tools the company never procured. Development teams stand up agents in a weekend. Vendors add AI features to software the company already licenses. Unknown AI is ungoverned AI, and the inventory gap is usually worse than leadership believes.
Three capabilities are required. First, a registry of every model, agent, prompt, retrieval pipeline, vector store, and workflow, functioning as the system of record for enterprise AI. Second, runtime telemetry that captures which model ran, who initiated it, what context entered, which tools were invoked, what came back, how long it took, and what it cost. Third, lineage that allows an organization to trace an output through the data, prompts, tools, models, and policies that produced it.
Sampling a small percentage of interactions is not control. When something goes wrong, “the model generated it” is not an explanation. Control means being able to determine why.
Once an organization can see the system, it has to define how the system should behave.
In control theory, the reference signal represents the desired state. For enterprise AI, policy is the reference signal. But a policy the runtime cannot read is guidance, not control.
Policies have to become declarative and machine-enforceable. They must specify which models may be used, which data they may access, which tools an agent may invoke, which actions require human approval, which outputs must be retained, which jurisdictions apply, and what cost, latency, quality, and risk thresholds are acceptable.
All of this has to be tied to identity. Every consequential action should be attributable to a person, service, application, or agent with scoped authority. Agents should not inherit developer credentials or operate through shared API keys that multiple teams keep in environment variables.
An agent without a defined identity and bounded authority is an insider threat with an API key.
This is where most governance programs quietly fail.
They document expectations. They convene review boards. They monitor a sample of outputs. What they cannot do is stop anything while it is happening.
Control requires mediation. The control plane has to sit in the execution path, where it can permit, block, redirect, constrain, escalate, or terminate. If a developer can bypass the control layer and call a model endpoint directly, the organization has a dashboard, not a control system. That is not a philosophical objection. It is an architectural one.
Runtime enforcement means four things in practice. Mediation ensures every model, tool, and data interaction crosses an enforceable control point. Routing and arbitration choose a model based on risk, cost, sensitivity, and task type rather than defaulting to whatever a developer selected eighteen months ago. Context governance determines what information is permitted to enter a prompt or workflow, which matters because many AI failures originate in excessive permissions or uncontrolled context rather than in the model itself. Explicit authority boundaries define when an agent may start, pause, approve, retry, terminate, or escalate.
A kill switch that works only because a human noticed is not a control.
Execution is not the end of the loop.
Verification requires evidence-grade records that allow an auditor, risk officer, customer, or regulator to reconstruct who initiated an action, what data and tools were involved, which policy checks fired, what the system produced, and what remediation followed.
It requires continuous evaluation because AI behavior changes. Models are updated. Prompts are edited. Data shifts. Users discover new uses, and attackers discover new ways to exploit the system. A system validated in March may bear little resemblance to the system operating in September.
It also requires economic attribution. Token metering tells you what was spent. It does not tell you whether the workflow produced value, and finance will eventually ask both questions in the same meeting.
Finally, it requires change control. Models, prompts, policies, data sources, and workflows need versioning and promotion through controlled environments. AI systems change too often for quarterly governance reviews to provide meaningful assurance.
A dashboard tells you something went wrong. A control system prevents the action, constrains its scope, routes it to a safer model, requires approval, or shuts the workflow down.
Observability without intervention is not control. Governance without runtime enforcement is not control. Audit records without feedback do not close the loop.
There is an obvious tension. Every control point introduces friction, and friction invites workarounds. Build this badly and the organization creates a shadow AI problem worse than the one it started with, because the fastest path to production now runs around the platform.
The answer is not fewer controls. It is controls that are fast, default-on, and easier to use than the bypass. If the control plane adds meaningful latency or requires a ticket, developers will route around it, and they will be right to do so.
My earlier research introduced the AI Controls Framework, a seven-layer reference architecture for deploying AI in enterprise and regulated environments. It covers the system of record, connectivity and access, data activation, workflow orchestration, policy and governance, delivery and embedding, and observability and continuous improvement.
This builds on my earlier research, Governing the Unpredictable: A Layered Reference Architecture for AI Systems Deployment
The framework explains where controls must exist across the architecture. AI Control Theory explains how those controls operate together as a continuous system. One is the architecture. The other is the operating logic. Enterprises need both.
A chatbot returns an answer. An agent retrieves sensitive data, calls tools, modifies records, sends communications, initiates transactions, writes code, and increasingly coordinates with other agents that no single team owns.
The failure mode is no longer a wrong answer. It is a wrong action executed at machine speed, potentially thousands of times before anyone notices.
That changes the requirement. The organization is no longer governing only what a system says. It is governing what the system can see, what it can decide, what it can access, what it can change, what it can spend, when it must stop, when a human must intervene, and what evidence it leaves behind.
As agency increases, control has to become more granular and more deeply embedded in execution. A governance committee cannot reliably catch these failures at machine speed.
The Questions Chief AI Officers (CAIO) Should Be Asking
None of these are model-development questions. They are questions of management control, architecture, accountability, and operational resilience. Most executive teams are not yet asking them with sufficient precision.
AI Control Theory should not rest on argument alone. It has to be tested against what actually happens inside organizations.
As part of my doctoral research at Claremont Graduate University, I am studying why enterprise AI deployments succeed, stall, or fail. I am collecting anonymous responses from people who have helped build, sponsor, govern, operate, or use AI systems at work. The survey takes about five minutes. It requests no name, email address, company name, or other direct identifier, and responses are analyzed in aggregate.
Please send it to colleagues with direct AI deployment experience. The broader and more diverse the participation, the stronger the evidence base will become.
This essay is the first in a series developing AI Control Theory for enterprise systems. Future articles will examine identity-bound agents, runtime mediation, machine-enforceable policy, economic attribution, and the architecture required to close the control loop.
AI will not become dependable infrastructure simply because the models become smarter. The models are already more capable than many of the workflows organizations have pointed them at.
AI becomes infrastructure when organizations can deploy it safely, repeatedly, measurably, and at scale. That is a control problem, and it remains open.
AI Control Theory is the discipline of turning organizational intent into observable, enforceable, and verifiable AI behavior.
The organizations that build this capability will move faster than those that do not, for the unglamorous reason that they will know what is running, what it is authorized to do, and whether it worked.
The next phase of enterprise AI will not be won solely by whoever has the best model. Access to highly capable models will become widespread.
It will be won by the organizations that can control them in production.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。