The Hidden Risk In Human-Like AI

Large language models (LLMs) promise a future of seamless, empathetic and intelligent customer service. But that promise comes with a critical flaw: They can be convincingly wrong.

Recent research conducted by SINTEF and boost.ai underscores this risk. Surveying 274 users on their perceptions of AI errors, the study found that not all mistakes are equal. The most damaging by far is factual inconsistency, when an AI hallucinates and provides incorrect information.

This type of error creates friction and shatters trust. When a customer asks about a mortgage rate, an insurance policy or billing terms, they are making decisions with real financial and legal implications. A wrong answer is a massive liability, and just like if an actual employee made the mistake, the company is on the hook. As one participant in the study put it: “It makes the chatbot redundant if I cannot trust the answer.”

That’s the reality organizations must confront. In the agentic age, accuracy isn’t a feature. It’s the entire product.

Why High-Stakes Environments Demand Control

In regulated industries, the tolerance for error is zero. The research highlights that users assess AI performance based on potential consequences. Errors tied to personal finances or contractual obligations are viewed as especially severe. A hallucinated response about insurance coverage or eligibility frustrates the user, and can trigger complaints, regulatory scrutiny or even legal action.

What’s more concerning is that many of these risks are invisible. AI doesn’t always fail loudly; it can also fail quietly. An answer that is partially correct but missing a critical condition can mislead just as effectively as an outright falsehood. If an AI tells a user they are covered but omits a key limitation, the outcome is the same: broken trust.

This is the “silent error” trap, and it’s where many AI deployments fall short.

Trust Requires Architecture, Not Optimism

Hallucinations are not edge cases, but an inherent characteristic of generative models. Hoping they won’t happen is not a strategy. Therefore, organizations must treat trust as a design constraint, not a post-launch fix. That starts with rethinking how AI systems are built.

Instead of relying on a single, all-purpose model, adopt orchestrated architectures and create entire systems that separate understanding from execution. In this model, generative AI plays a specific role: interpreting user intent. It identifies what the customer is trying to accomplish, but it doesn’t necessarily generate the final answer. Execution is handled by specialized agents. And in high-risk scenarios, those agents need to be rule-based and operating within predefined, compliance-approved frameworks.

This distinction matters. A rule-based agent doesn’t “guess.” It follows a verified process, and it cannot invent an answer or omit a critical clause. By separating these layers, you will dramatically reduce the risk of factual inconsistency while still benefiting from the flexibility of generative AI.

3 Strategies To Build Trust In The Agentic Age

If trust is the asset, then governance is how you protect it. Organizations looking to scale AI responsibly should focus on three core strategies:

1. Design For Accuracy First, Not Fluency

Human-like responses are compelling, but they can be misleading if they aren’t grounded in verified data. Prioritize systems that ensure correctness, even if that means sacrificing conversational elegance in high-stakes moments.

2. Orchestrate, Don’t Centralize

Avoid the temptation to rely on a single model for every task. Use orchestration to route requests to the right agent. Generative and conversational where flexibility is needed, rule-based where precision is required.

3. Build For Recovery, Not Perfection

Errors will happen. What matters is how quickly and transparently they are resolved. Ensure users have clear paths to escalation, whether that’s another agent or a human. Trust is often preserved not by avoiding failure, but by handling it well.

Trust Is The Real Competitive Advantage

There’s a misconception that AI advantage comes from access to better models. In reality, most organizations will have access to similar capabilities. The difference will come down to trust. Customers will gravitate toward companies whose AI systems, built on top of the underlying LLM technology, are reliable, transparent and aligned with their needs. And regulators will increasingly scrutinize those that aren’t.

The agentic age isn’t just about automation. Organizations are asking AI to act on behalf of their customers, and by extension, on behalf of their brand. That’s a profound shift, because when an AI agent speaks, it doesn’t sound like a machine. It sounds like a member of your organization. And in that moment, trust is the output that matters most.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?