Mark Hull is CEO of Exceeds AI where he helps CTOs and engineering leaders understand how AI and agentic workflows impact outcomes and ROI.
getty
For two years, technology leaders have had a buffer: AI was new, pilots were experimental and adoption itself was the goal. That window is closing fast. Every AI line item now has a sponsor in finance and a board director with sharp questions. Boards want to know what they're getting for the money.
In April, Uber's CTO acknowledged the company had used its entire 2026 AI engineering budget in four months. Claude Code adoption inside the 5,000-engineer organization jumped from 32% to 84% in a quarter, with some engineers running $500 to $2,000 a month in API spend. MIT's 2025 NANDA report found 95% of corporate generative AI initiatives delivered no measurable P&L impact across 300 surveyed deployments.
If you can't answer the ROI question, your budget and your credibility are both at risk.
Last quarter, I sat with the head of platform engineering at a mid-market SaaS company. Her dashboard told a clean story: AI adoption was at 78%, cycle time was down 22% and developer satisfaction was green. The codebase told a different story. Pull requests had grown larger. Test coverage on AI-assisted modules was drifting down. The same three patterns kept appearing across services because the assistant defaulted to whatever the nearest neighbor file looked like.
There are three key needs that surface in all of my customer conversations. First, you need a defensible AI ROI answer for the board. Second, you need visibility into how engineers run AI sessions. And finally, you need a metric for AI-produced code that survives. Most measurement stacks deliver none of the above.
Most leaders default to efficiency metrics because they're easy to instrument: lines of code, pull requests opened, cycle time and sentiment scores about how much faster the team feels. A sharp CFO will eventually call them leading indicators of nothing.
You can ship more pull requests and produce worse software. The 2025 DORA State of AI-assisted Software Development Report reveals some startling numbers: Bugs per developer are up 54%, incidents per pull request are up 242%, median review time is up 441% and 31% of pull requests are now merging with no review at all.
Your activity dashboards aren't lying. They're measuring inputs while your board has started asking about outcomes.
Not sure where to start? The following are key metrics that I'd urge all CTOs to start measuring:
• Proficient Use, Not Seat Counts: A 90% activation rate means little if your engineers are using AI as fancier autocomplete. A well-run session uses a deliberate harness, retrieved context, a multi-step plan, the right model for the task and a verification loop. A YOLO single-prompt session registers identically on your dashboard. Inspecting sessions paired with the resulting code is how you separate a coaching opportunity from a systemic gap.
• Outcome Quality, Evaluated Against The Code Itself: Was the AI-assisted code more defective? Less maintainable? More duplicative? Did it pass review faster because it was better, or because reviewers gave up? CFOs need to focus on cost-per-outcome: dollars spent per piece of AI-assisted code that ships and survives 30 days in production.
• Learning From Your Heroes: Inside every engineering org, 10 to 20 people are getting two or three times the lift everyone else is. Find them. Codify what they do. Stanford researcher Yegor Denisov-Blanch's 2026 study of 120,000 developers saw AI productivity gains of 30% to 40% on low-complexity greenfield tasks but 10% or less on high-complexity legacy code. The variance is task selection, not the tool.
• Expanded Ambition: The real return on AI comes when your team is willing to attempt what they'd have rejected a year ago. This could lead to new deliveries for shareholders and customers once your team is freed from toil.
Most measurement approaches look at the wrapper around the work, not the work itself. Surveys capture how the work felt. Cycle time captures how fast it moved. Neither tells you whether the result was any good. The same Stanford research linked above found a 30-percentage-point gap between developers' self-reported and measured productivity. That number should give any survey-based ROI report pause.
Agentic workflows make the gap wider. Metadata, logs and API pulls confirm a token was spent and a call was made. What the agent did, how the engineer steered it, where the session broke down, whether the output got merged: None of that shows up in the wrapper data. If you want to know, not guess, what individual engineers are doing inside agentic workflows, you have to inspect the workflows themselves.
There are three questions your team needs to start answering: What did your team produce that they couldn't have before? Where's your code quality going? And what's your next ambition?
The "we’re still learning" era has ended. I believe the leaders who design their systems for outcomes will spend next year delivering healthy numbers to their boards.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。