A recent paper about AI agents in production revealed something fascinating:
Most real-world “AI agents” are not autonomous systems.
They’re heavily constrained workflows with:
- limited autonomy,
- strict tool access,
- human supervision,
- deterministic guardrails,
- narrow scopes.
In other words:
Production AI systems are far less autonomous than AI discourse suggests.
And honestly, that makes complete sense.
Because modern software was specifically designed to reduce ambiguity.
LLMs reintroduce it everywhere.
The hidden purpose of user interfaces
We usually think of UIs as convenience layers.
Buttons.
Forms.
Menus.
Tabs.
But that’s not their real purpose.
The real purpose of a UI is constraint.
A good interface:
- narrows the space of possible actions,
- reduces interpretation,
- exposes capabilities,
- prevents invalid states,
- guides user behavior.
In other words:
Good software reduces ambiguity.
That rigidity is not accidental.
It’s architecture.
Natural language removes constraints
Now compare that with an AI agent.
Instead of clicking:
Approve invoice
you type:
“Can you handle the invoice issue from last week?”
This sounds more human.
But technically, it’s dramatically worse.
The system now has to infer:
- what “handle” means,
- which invoice,
- which workflow,
- which permissions,
- which side effects are acceptable,
- how confident it should be.
We replaced explicit intent with probabilistic interpretation.
And unlike traditional software bugs, LLM mistakes are often:
- plausible,
- silent,
- partially correct,
- hard to detect.
Which is exactly what makes them dangerous.
The paper accidentally exposes the core issue
The study is interesting because it quietly contradicts much of the “fully autonomous agents” narrative.
The paper found that most production agents:
- execute relatively short workflows,
- rely heavily on human validation,
- use simple orchestration patterns,
- operate inside constrained environments.
Why?
Because unrestricted probabilistic systems are operationally unstable.
Once AI moves from demos into production, companies rediscover why software engineering evolved around:
- constraints,
- validation,
- permissions,
- deterministic state,
- explicit workflows,
- auditability.
The more critical the system becomes, the more autonomy gets reduced.
That’s not a temporary limitation.
It may actually be a fundamental architectural reality.
Most software is not generative
This is where AI discourse often disconnects from reality.
The majority of the software industry is not about generating things.
It’s about maintaining coherent state.
ERPs.
CRMs.
Accounting systems.
Identity systems.
Supply chains.
Payments.
Compliance systems.
These platforms exist to guarantee consistency.
Not creativity.
Which explains why fully conversational interfaces often feel awkward once you move beyond demos.
To use them effectively, users frequently need:
- a mental model of the system,
- knowledge of available capabilities,
- understanding of workflow implications,
- awareness of hidden constraints.
Ironically, natural language interfaces can increase cognitive load instead of reducing it.
AI works best where ambiguity is acceptable
This is why the most successful AI products today are mostly:
- copilots,
- assistants,
- drafting tools,
- semantic search systems,
- developer tooling,
- workflow accelerators.
Cursor.
GitHub Copilot.
Notion AI.
Figma AI.
Notice the pattern:
These systems operate in environments where:
- multiple outputs may be acceptable,
- humans remain in the loop,
- approximation is tolerable,
- iteration is expected.
That’s very different from transactional software.
The future is probably hybrid
I don’t think software becomes “just chat”.
I think the winning architecture looks more like this:
Deterministic systems + Probabilistic interaction layers
The core business logic remains structured and reliable.
AI becomes:
- an interface layer,
- a semantic layer,
- an orchestration layer,
- a cognitive acceleration layer.
Not a replacement for deterministic architecture.
And interestingly, that’s exactly what we already observe in production systems today.
Not autonomous agents.
Constrained probabilistic interfaces.
Maybe AI is really an interface revolution
I increasingly think LLMs are less important as reasoning engines than as interface technology.
For decades:
humans adapted themselves to software.
Now:
software starts adapting to human expression.
That’s genuinely transformative.
But removing constraints entirely may turn out to be one of the worst possible ideas for complex systems.
Because constraints were never the enemy.
They were the thing making software reliable.