Jeremy Ung is Apptio’s Chief Technology Officer, leading the Engineering, Operations, and Global Support teams. With almost 20 years of experience in software engineering across development, product management and engineering management, Jeremy ensures his team is modernizing and transforming development processes, practices, and technologies to enable greater scale, agility, and customer value. Here he discusses business considerations when measuring return on AI investments.

Artificial intelligence (AI) is nothing new, quietly supporting us from behind the scenes for years – from efficiently routing us to and from work to detecting fraudulent charges in our bank accounts. However, recent advancements in large language model (LLM) capabilities have catapulted AI into the spotlight. These achievements have flooded our news feeds with impressive feats and produced a burgeoning crop of AI-centered startups. Despite this AI evolution’s newness presenting several complexities and strategic considerations, optimism has outpaced reticence, and established enterprises are collectively investing billions to leverage the technology’s potential.

Findings from our recent research report Executive Insights on Tech Investment Decisions support this optimistic view. Three-quarters (73%) of tech leaders implementing AI are highly confident in relying on it for business decisions, and nearly 90% of organizations plan to implement AI and/or dedicate portions of their budgets to AI over the coming months.

While business futurists evangelize AI’s potential, it’s often without delving into the nitty gritty details of business value and returns – and sometimes, there can be a bit of a disconnect between future promise and current reality. It’s not too surprising, given how frothy the market has become. AI absolutely offers exciting possibilities and has the potential to solve a myriad of use cases, but for pragmatic tech leaders, it’s best to approach this innovation with a complete understanding of ancillary impacts and considerations to get the best return on investment.

Level-Setting on AI: The Challenge of Data Quality and Training Models

Generative AI and LLMs have become part of the modern nomenclature with free-to-use models like OpenAI’s ChatGPT, Anthropic’s Claude, and Google’s Gemini. While these off-the-shelf models are incredibly impressive, they require more fine-tuning to support the enterprise. Significant model training is required to produce valuable, accurate results tailored to a particular business and its industry. Furthermore, outputs must be constrained to prevent biases and stay within ethical and legal guardrails. TechTarget’s recent Generative AI in IT Operations report echoed this sentiment: “Organizations have healthy concerns about bias, fairness, trainability, and accountability in terms of challenges when integrating AI into IT operations.”

Data quality is another significant factor in AI development, epitomized by the old computer science aphorism, garbage in, garbage out. For AI models to be effective, they require meticulously cleaned and well-curated data, a significant hurdle given the disparate and often incomplete datasets within enterprises. Poor data quality not only leads to ineffective outcomes but can also cause AI models to produce misinformation or “hallucinate.” The consequences of this can be dire. Think of the negative impact of AI-generated misinformation while trying to respond to an emergency, not to mention applications related to healthcare or finance. Despite these challenges, with the right data, LLMs and machine learning algorithms hold immense potential to accelerate businesses in a multitude of areas.

The True Costs of AI – Infrastructure, Data, and More

As the hype train reaches full speed, we see companies throughout the market looking for ways to fold AI into their existing products. But before we jump to the conclusion that AI is table stakes, it’s essential to consider the costs and tradeoffs.

AI doesn’t come cheap, relying on extensive infrastructure and computing resources to crunch the massive amounts of data required to train and run models. Additionally, data storage costs, data transfer fees, and expenditures related to model monitoring and maintenance all add to the bill. While Forrester foresees a 36% compound annual growth rate in genAI from 2023 to 2030 – they also see AI platform investment tripling to meet demand, including greater demand for more AI infrastructure.

But it’s not just hardware and software; it’s human capital, too. As mentioned above, clean data is a critical component of developing successful AI, so companies need people to clean and curate data for that purpose. Likewise, there will need to be ongoing human review of outputs to ensure quality and accuracy. Plus, if advanced fine-tuning of models via vector databases and post-processing of model results is required, workers with that skill set are scarce – and expensive.

The use of customer data is another sticky point. While it represents a treasure trove of potential insights, and companies may be eager to leverage it, there’s a serious question about whether they have the right to this data. Overall, when it comes to data, numerous privacy and security implications need to be resolved before getting underway.

Regarding mitigation for the potential impact of hallucinations and misinformation, Forrester’s report also predicts that a major insurer will offer a policy specific to these AI risks. This cost will provide indemnification from the resultant negative consequences, such as reputational damage – not to mention equally significant economic and operational impacts.

Finally, governance is another major consideration. Forrester sees 40% of enterprises proactively investing in AI governance for compliance. As regulatory enforcement of AI ramps up in the US, EMEA, and APAC, we’ll see greater attention paid here in the form of policy changes and a workforce of in-house and external support to ensure preparedness for evolving regulations and compliance issues.

Thinking About AI Investments Strategically

The urge – and pressure – to dive headlong into AI is strong right now. And it’s easy to envision a future wherein the majority of our tools and products are infused with this promising tech. We’re certainly at a watershed moment where publicly available LLMs’ impressive performance has captured the imagination of companies and their customers. No company wants to be left behind, but it’s essential to remember that a great deal of this technology and its application is experimental. So, how do we approach this innovation prudently?

Customer Goals and Needs

Start by keeping your customers’ goals and unique needs top of mind. Is the application of AI under consideration one that will actually help my customer with their workflow or make their job easier? Also, keep in mind that AI can have performance impacts. Unless closely monitored and compensated for, AI’s resource consumption can drag application performance down. Is an LLM the right approach, or would a machine-learning model be faster? Be sure to keep in mind whether AI is adding value and ensures that customer trust and confidence is at the fore.

Unit Economics of AI

It’s also critical to understand the full unit economics of AI. Factoring in all the aspects of infrastructure and workforce investment needed – plus the governance, regulatory, and insurance pieces – a lot of analysis is required to make funding decisions. Only with a clear view of all correlated costs and ongoing evaluation of impact can you ensure an ROI on AI projects.

Taking a Measured Approach

Realize that much of this innovation is still in the experimental and exploratory stages. While low-hanging fruit use cases, like customer support bots, show early signs of promise, more advanced use cases will take some time to refine and determine where the more significant areas of impact may lie. In many instances, it may make sense to take a measured approach, getting the fundamentals in place, laying the groundwork for intelligent, strategic decisions regarding AI projects.

Making Technology Investment Decisions

The goal of any technology leader is to make tech investment decisions that deliver value to the enterprise. Whether we’re exploring AI, cloud, or any other innovative technology, that objective never changes. Approaching these decisions takes considerable thought – as evidenced above. To see how other technology leaders approach decision-making like this, check out our latest research report, Executive Insights on Tech Investment Decisions. With over 1,700 respondents, we uncovered insights into how tech executives make data-driven decisions, the importance of flexibility and frequency in decision cycles, and considerations when prioritizing budget allocations for AI and other accelerating fields, such as cloud and cybersecurity.