As the CEO of Arango, Shekhar Iyer leads the company’s mission to make enterprise AI contextual, scalable and trusted.
getty
As organizations shift from AI pilots to production, I’ve observed that many leaders are focusing on metrics that are relatively easy to measure, such as GPU utilization, token consumption and infrastructure spend. But in my experience working with enterprise AI and data leaders, there are hidden costs that need to be measured and monitored closely.
The hidden cost of agentic AI is the engineering effort required to continuously rebuild real-time business context across fragmented systems. The urgency is growing quickly. According to Gartner, “only 17% of organizations have deployed AI agents to date, yet more than 60% expect to do so within the next two years.”
In light of that, it’s no surprise that in my conversations with enterprise AI and data leaders, many are grappling with these questions at their organizations:
• “Why are our AI costs rising faster than business value?”
• “Why are highly skilled teams spending more time stitching together systems than improving outcomes?”
• “Why do AI pilots that looked promising suddenly become difficult and expensive to scale?”
In my view, AI success ultimately depends on solving data and context architecture challenges. Until that changes, many teams risk being trapped—spending too much time and money on data plumbing rather than building features that generate business value.
When it comes to AI implementation, compute costs are only part of the equation. AI infrastructure costs are visible. AI tokens are akin to a taxi meter. When AI systems generate more prompts and outputs, organizations receive bigger bills.
But AI implementation also has a hidden cost: the data engineering effort required to make AI agents work effectively. From my observations, many organizations operate across numerous disconnected enterprise systems for their day-to-day operations. The data needed for AI agents to operate reliably is locked inside those fragmented systems.
Data plumbing—stitching together systems and rebuilding context across them—can consume significant engineering effort that drives up data infrastructure, integration and operational costs. Several leaders I’ve spoken with estimate that maintaining data architecture consumes 70% of their overall AI investment. They’ve also told me that inefficient data architectures have driven their token costs substantially higher.
For many organizations, it’s not feasible to run AI agents on data from a single isolated system. Doing so could result in inaccurate, misleading or incomplete outputs. For instance, if an e-commerce clothing retailer uses an AI agent for customer service, it needs to pull in data from its customer relationship management platform and its inventory, shipping and order management systems to provide accurate responses about purchases, delivery timelines, product availability and returns.
Gartner predicts that more than “40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls.”
In my experience, many organizations began their AI journeys by experimenting to demonstrate the art of the possible, often without a long-term production architecture in mind. I’ve seen teams frequently start with vector databases for embeddings, then layer additional tools, pipelines, graph technologies and integrations over time as requirements evolve.
As complexity increases, AI pilots often lack the necessary business context to deliver trusted, explainable results. This problem often intensifies once business leaders decide to scale their AI solutions and move them into production. At that point, it may be necessary for engineering teams to go back to the drawing board and rebuild parts of the data architecture, a process that requires significant time and effort. Over time, more engineering effort goes into maintaining fragmented pipelines and integrations than building features that deliver business value.
Rebuilding business context across fragmented systems also introduces more latency, higher compute requirements, greater operational overhead and increased token consumption. As AI agents scale, these inefficiencies tend to compound, chipping away at the organization’s ROI.
In my view, the fastest way to reduce the time and money costs associated with AI implementation is to eliminate misallocated effort. To reduce it, I recommend that AI and data leaders take a few steps.
At the core, I advise leaders to plan their data architectures for production scale from day one.
To reduce systemic rework, leaders should measure engineering effort directly: How much time are their teams spending rebuilding context across fragmented systems? If leaders find that their team members are spending too much on data plumbing, that’s a sign that too much of the organization’s focus is going toward stitching systems together rather than improving business outcomes.
When leaders address data and context architecture from day one of their AI design and implementation, or as soon as they notice their teams spending significant time stitching together fragmented systems, they’re more likely to reduce their engineering, maintenance and operating costs—and generate successful outcomes.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。