Ian Kraskoff is CEO and founder of Cloud Humans, an AI platform that automates customer support for all written channels, including WhatsApp
getty
Last quarter, I spoke with the head of customer experience (CX) at a scaling SaaS company. Six months after deploying their AI agent, their automation rate had reached 68%, while their net promoter score (NPS) had dropped 11 points over the same window. The issue was not the model but rather that nobody owned the architecture. Intents were inherited from a legacy ticketing taxonomy, the knowledge base had never been audited and escalations landed in a generic queue without context. Each layer had a different owner, which meant no owner at all.
This pattern is not an exception. Gartner predicts that by 2027, half of the companies that cut customer service staff because of AI will need to rehire. The MIT NANDA "State of AI in Business 2025" report found that 95% of enterprise AI pilots fail to produce measurable P&L impact. The pattern is consistent: Companies don't treat AI deployment as a system design problem.
In my previous article, I argued that CX is becoming an engineering discipline. This piece goes deeper into what that discipline looks like, from the layers to who owns them and how they fit together.
Before any prompt, the CX engineer must understand the business's objectives. A refund request from a high-ARR account and a $5 DTC customer look identical in the queue but have completely different business implications.
The domain problem isn't new. A McKinsey and University of Oxford study found that the top predictors of success were not technology choice but strategy, stakeholder alignment and talent, with failure on those dimensions accounting for about half of all cost overruns.
For every type of customer question, CX engineers decide which mechanism will resolve it. Rather than asking "What can we automate?" they ask "What do we not want the AI to handle?" Everything else becomes a design problem.
Some questions are resolved by content, some by a specialized micro-agent, some by access to back-office systems and some by a multi-agent setup that mimics human judgment.
This is where hallucination is won or lost. Research on grounded summarization shows top-tier LLMs hallucinate as little as 0.7% to 1.5% of the time when properly grounded in source material, but rates climb sharply as documents become longer and more complex.
The common trap is treating knowledge as a dumping ground: Articles written for humans get indexed wholesale, duplicates pile up and contradictions sit unresolved. Good knowledge architecture requires atomic facts, versioned entries, clear ownership and periodic audits.
If Layer 2 is the blueprint, Layer 4 is the plumbing. The orchestrator routes conversations to the right specialized agent, passes full context between agents, monitors confidence scores and decides whether to continue, retry or escalate. Research on LLM-based agents shows hallucinations accumulate across reasoning steps in multistep workflows, which is why disciplined handoff and confidence logic are not design preferences.
A Qualtrics study found that "nearly one in five consumers who have used AI for customer service saw no benefits." Cisco reported that its Webex AI Agent reduced escalations by 85% for one large customer by making escalation trustworthy. Good escalation is a feature, not a failure.
Automation and quality rates should always be paired. Klarna is the canonical cautionary tale. In early 2024, the company announced that its AI chatbot was handling two-thirds of support conversations and doing the work of 700 agents. By mid-2025, the CEO publicly admitted that the AI-first strategy had produced "lower quality" output, and the company began rehiring humans. Automation without quality rate is vanity.
Quality requires enough manually reviewed conversations to establish what "good" looks like. These, then, calibrate an LLM-as-judge pipeline, enabling you to scale. Well-calibrated LLM judges can reach over 80% agreement with human evaluators, comparable to inter-annotator agreement between humans themselves.
But calibration is not a one-time thing. Reviewers must continuously correct disagreements, feeding those corrections back into the evaluator. Automated eval alone drifts, and manual audit alone does not scale, so the discipline is to run both in parallel, forever.
Good continuous optimization requires two specific competencies. The first is experimentation literacy—understanding A/B tests, power, sample size and when you have enough signal to act. Without this, teams either ship changes on noise or freeze because results never look "significant" in a small weekly sample. The second is knowing where to tune. A drop in quality can come from the content layer, the micro-agent layer, the orchestrator or cross-agent characteristics.
This layer also requires an uncomfortable competency: humility about what your company can actually sustain. The MIT NANDA report found that purchasing AI tools from specialized vendors succeeds about 67% of the time, while internal builds succeed only one-third as often. Costs follow the same pattern: Custom enterprise AI runs $100,000 to $500,000 up front plus $5,000 to $20,000 per month in maintenance, with 65% of total costs materializing after deployment.
The CX engineer has to be honest about whether the company has the engineering depth, roadmap patience and long-term commitment to own this stack forever. If it isn't, partner with a specialized vendor for a fraction of the cost of building internally. That is the same discipline that led companies to stop running their own data centers and writing their own CRMs.
None of these layers works in isolation. Strategic CX knowledge without resolution architecture produces vague systems. Resolution architecture without knowledge structure produces fluent hallucinations. Knowledge without orchestration produces AI that knows everything and can do nothing. Orchestration without monitoring produces fast roads to bad outcomes. The CX engineer is the role that holds the layers together because no one else in the organization is positioned to see all six at once.
In the next 18 to 24 months, I believe CX engineering will stop being a framework and start being an org chart convention, the same way DevOps did a decade ago. The companies that start now may avoid having to rehire in 2027.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。