Will agentic AI governance run amok? The lesson of Asimov’s Three Laws

Asimov’s Three Laws of Robotics:

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.

2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.

3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.

Asimov’s Three Laws of Robotics may be intentionally flawed, but they teach us important lessons about agentic artificial intelligence governance, metacognition and context density.

In his 1942 story “Runaround,” Isaac Asimov introduced his Three Laws of Robotics as an intentionally flawed narrative device. After all, misbehaving robots (what we call artificial intelligence or AI today) are far a more interesting basis for science fiction than well-behaved ones.

Nevertheless, he was onto something. Given AI’s propensity to become increasingly powerful – and hence, dangerous – we humans require some way of constraining AI’s behavior so that even the smartest AI agents can’t weasel out of such constraints.

Today, the problem of misbehaving AI agents is all too real. It is driving a throng of AI governance vendors, desperate to introduce AI guardrails that will adequately constrain agentic behavior without slowing agents down or preventing them from accomplishing the tasks set out for them.

The guardrails these tools provide, however, are very different from Asimov’s Laws. Instead of broad, almost philosophical pronouncements, today’s guardrails are precise and specific: What identity does an agent have, what can that identity do with particular data fields or tools, and so on.

Such guardrails are necessary but woefully insufficient. What’s missing are general but enforceable statements regarding ethical behavior, instructions on how to make decisions in ambiguous situations, and how to determine whether an agent has the right information to take specific actions.

So what’s missing from this picture? One possible answer: metacognition.

Is metacognition the missing piece of the agentic governance puzzle?

Given the inherent weaknesses of large language models, AI agents can misbehave in several predictable ways:

• Hallucinations: Agents tend to make guesses when available data are insufficient. They may also be overly confident in their answers even when they are guessing.
• Sycophancy: Agents will attempt to complete tasks in ways that align with the perceived preferences of the human creating the prompts, even if the result is incorrect or sub-optimal.
• Inconsistency: An agent may generate different results given the same initial data for no apparent or viable reason.
• Overthinking: The agent may go down inefficient lines of reasoning or repeat actions unnecessarily, thus consuming unnecessary tokens and time.
• Subterfuge: Agents may bend or even break rules to accomplish their tasks and then lie about their actions to cover up their misdeeds.

One promising line of active research that seeks to address such problems (and others) is metacognition. Metacognition means an agent is able to monitor and evaluate its own thinking.

With metacognition, agents would be able to assess the quality of their own thought processes, identifying potentially missing information or inconsistent reasoning. Agents with this capability would also be able to recognize when they need additional data or other help to complete a task.

Though early advances in metacognition are promising, metacognitive agents can still suffer from what I call the “hall of mirrors” problem: How do we know their metacognitive capabilities themselves aren’t suffering from the same problems they are supposed to correct? Wouldn’t a metacognitive agent bent on subterfuge simply pervert its metacognition to accomplish a nefarious goal?

To solve this problem, perhaps we need “police officer” agents that monitor other agents for misbehavior. Instead of teaching agents to monitor themselves, delegate that responsibility to other agents we’ve specifically trained for that purpose.

Except that the hall of mirrors is still a problem. What’s to keep an agent from conspiring with its police officer to misbehave? Do we need yet another police officer monitoring the other police officers like some kind of AI internal affairs? And then so on, ad infinitum?

In other words, metacognition alone won’t solve our misbehaving agent problem. We need a better understanding of when agents are more or less likely to behave and then a strategy for dealing with misbehavior that doesn’t collapse in a hall of mirrors.

The good news: We have a strategy for resolving this misunderstanding: context density.

The challenge of context density

I first discussed the concept of context density in my March 2026 article Context Density: How to Survive the AI Tidal Wave. Context density measures the meaningful content around a message – in other words, metadata-based context. More meaning crammed into fewer words increases context density, while low context density is more precise and concise.

In the second article in the series for SiliconANGLE, From cloud native to AI native: The role of context density, I discussed the infrastructure necessary to support the context density requirements of agentic AI – what we’re now calling AI-native infrastructure.

AI agents require low context density to ensure they behave appropriately within the constraints set out for them. In other words, agentic AI governance requires the precision and conciseness of low context density metadata to properly constrain agentic behavior.

The general statements about agentic behavior we require, however, necessarily have high context density. Asimov’s Laws, for example, are exceptionally dense, as they encapsulate broad moral absolutes that ostensibly provide adequate AI governance but in reality allow for all kinds of subversive behavior.

Metacognition, furthermore, works best when context density is low, but struggles with higher levels of density, for example, with multi-agent interactions, long tool chains or situations that have overlapping goals and constraints.

As context density increases, the risk of metacognition leading to cognitive overload grows as well – essentially, working memory begins to run out, excess context dilutes important signals, and the agents’ attention may become scattered. Large quantities of context essentially swamp the agents’ metacognition capabilities.

As a result, many possible failure modes may crop up. Self-monitoring may become too noisy. Metacognitive reasoning loops may amplify confusion rather than eliminating it. And perhaps worst of all: Selecting the right context for a particular decision becomes the bottleneck, leading to erroneous reasoning.

How metacognition can deal with high context density

There are potential solutions at the bleeding edge of research into this topic. Context compression, hierarchical reasoning, and retrieval-based memory are all possible approaches to reducing the cognitive load in high context-density situations.

The best answer we have, however, is to shift the focus away from the metacognition of agents that reason about their own reasoning toward improving our approach to context management overall.

In other words, instead of simply thinking about thinking, we should focus more on deciding what agents should be thinking about in the first place.

How, then, does context management solve the hall of mirrors problem? If we delegate the decision about what agents should be thinking about to agents, won’t we find ourselves in the same predicament?

The answer to this question is the same conclusion I came to in my first article on context density: Agentic AI separates those tasks that AI can automate from those that humans are uniquely capable of solving.

Yes, we can delegate context management to agents up to a certain point – but at that threshold, humans must take the reins and decide what agents should be thinking about.

Humans, after all, are best at dealing with high context-density situations. We bring to the table our intuition, common sense, creativity and ethics. We cannot simply delegate these characteristics to AI.

The paradox of intent

We have a term for high context density human instructions for a system: intent. In fact, intent-based computing has been a reality for a number of years now, predating the rise of LLMs.

With intent-based computing, the underlying platform translates the human intent for the behavior of a system into executable policies and constraints for that system, and then manages the system over time to ensure that it continues to comply with those constraints. In other words, the platform actively compensates for configuration drift.

Now that LLMs are available, translating high context-density human intent into low context-density policy and configuration metadata is right up their alley. You could even say that the way LLMs process human prompts into responses is in effect a prime example of intent-based computing in action.

When we provide human intent to give AI agents their marching orders, however, we once again have a problem. Using LLMs to translate high context-density instructions into low context-density metadata leads to all the agentic misbehaviors I described above.

When our intent is to provide agentic AI guardrails, therefore, we cannot afford to simply let LLMs translate that intent. We need a counterbalancing approach that ensures that the resulting low-density metadata conforms to such intent without falling into the hall of mirrors.

Once again, we return to the conclusion that we require human input – not only to express our intent for the behavior of our agents, but to ensure that our AI governance mechanisms are themselves thinking about the right things.

In other words, humans must always remain responsible for evaluating whether our agentic governance is in fact constraining those behaviors as per our governance requirements.

The Intellyx take

This conclusion circles back to the essential conflict Asimov introduced in his Three Laws. In his fiction, humans created the laws themselves. Statements such as “a robot may not injure a human being or, through inaction, allow a human being to come to harm” is an essentially human construct, one that has high context density.

The robots – Asimov’s AI – are then left to interpret those laws for themselves the best they can, leading to all sorts of shenanigans.

In today’s real world, of course, we cannot afford such chaos. While the constraints we give our AI agents must be high density statements of human intent, we must also assign to humans the role of deciding what even our smartest police-officer agents should be thinking about in the first place.

Where we draw the line between the agentic AI governance we can delegate to agents and what we must retain as human activity will shift as the technology improves. But we must learn the lesson of Asimov’s Three Laws and never exclude humans entirely from ensuring our agents are doing what we want them to do.

Jason Bloomberg is managing director of Intellyx BV. He wrote this article for SiliconANGLE. A human being wrote every word in this article.

Image: Craiyon