Your inbox, upgraded.

Receive weekly engineering insights to level up your leadership approach.

Tokens burned has become the easiest number to point to. It’s objective, it’s automated, it scales across thousands of engineers, and it gives leadership a dashboard. It’s easy to measure. It’s also easy to game.

“Measuring AI adoption and productivity gains is hard, and especially if you have to do it at an individual level. Token usage definitely isn’t the right metric. You could just use tokens to run your OpenClaw!” says Ankit Jain, founder of the Hangar, a community of senior engineers and engineering leaders focused on developer experience and solving productivity challenges at scale.

Tokenmaxxing is going back to the pre-DevOps Research and Assessment (DORA) and to ‘measuring lines of code’ era, he adds. DORA metrics are a set of five software delivery performance metrics that provide an effective way of measuring the outcomes of the software delivery process.

However, DORA metrics were never designed to evaluate individual developer output. They work as multiple metrics, not just one, Jain says. If you game one metric, it’s going to break down other metrics.

When organizational trust is low, managers reach for data to justify headcount and budget decisions, and individual productivity measurement follows. That’s how DORA gets misapplied as a personal scorecard, and the same dynamic is now producing tokenmaxxing leaderboards.

“Organizations need to figure out how much outcome AI tools are driving, and that’s a hard problem to measure,” Jain says. The solution, he argues, is combined metrics: “We have to come up with a set of metrics versus just one. Tokens used, though, has to be one of them.”

One of the more interesting attempts to move beyond token counting comes from a staff engineer at a FAANG company we spoke to, who has been building an AI adoption measurement framework at their organization.

The framework is inspired by Steve Yegge’s ‘executive chef’ model that compares software engineers to chefs in a restaurant kitchen. AI agents are engineers’ sous chefs, line cooks, and prep cooks. Just like executive chefs don’t do all the chopping themselves but decide what goes on the menu, develop the recipes, and taste everything before it’s served, engineers orchestrate agents and own the outcome.

They have adapted Yegge’s original nine levels to four, ranging from basic interactive tool use to orchestrating multiple autonomous agents. The levels do not measure which tools engineers use or how many tokens they burn, but rather the degree of cognitive work being delegated to AI over time, and how effectively.

“A developer at level one is using AI for quick one-shot queries. At level four, they’re writing detailed specifications, orchestrating multiple agents, and shifting quality assurance left toward spec review rather than code review,” they say.

The framework also revealed a counterintuitive proxy signal: as engineers progress through the levels, they file fewer bugs because they resolve issues inline rather than queue them. A declining bug backlog growth rate becomes a lagging indicator of genuine AI maturity – not something you’d normally think to instrument for, but meaningful once you know to look.

The data also surfaced a bigger problem: the level progression is not linear or obvious. Engineers at level one can’t easily imagine level four. There is a real adoption challenge that token dashboards detect but can’t diagnose: engineer resistance.

“There is a genuine pushback of ‘I don’t want my job to be an orchestrator of agents,’ which is an identity concern, not a capability one, and a much harder thing to measure or address. Any honest framework for AI adoption has to account for where an organization sits on that curve,” they say.

Emily Nakashima, SVP of engineering at Honeycomb, says there is a lot of fear among people at software companies (not just among engineers) when the question of measuring AI impact and productivity is raised. People worry about what that means for their jobs.

“We did a top-down founder memo last summer, saying that we really believe in this new technology, we really want people to spend time experimenting and learning about it, and that we should all try to 2x our impact with AI over the next year. The first question we got from engineers was ‘how are we going to measure this?’” she says.

Rather than building formal frameworks, Nakashima says they deliberately de-emphasized measurement, relying instead on self-reporting.

“I really worry about companies trying to 2x or 3x their token spend, because there are ways engineers can do that that return no value to the company. We actually get a lot of value out of self-reporting. For that to work well, you have to have a relatively high-trust organization. What engineers on my team give back in terms of self-reporting actually aligns pretty well with what’s seen in their work. When it’s working, you can see it, and these measurement questions go away a little bit.”

Before organizations can measure whether AI is making engineers more effective and where the skill gaps are, managers need to know whether they’re using it at all. Token usage is a good proxy for AI adoption, but it’s a terrible metric for AI productivity.

Angie Jones, former VP of engineering, AI tools, and enablement at Block, confirms that measuring token usage was helpful when they were early in their journey and she needed to track adoption.

“After adoption was clear, I threw it out as it did nothing to measure developer productivity. I wouldn’t be surprised if we see the opposite trend next year, aiming for efficient usage of tokens as opposed to celebrating burning them at expensive rates.”

However, there is no consensus yet on what to measure instead. Jain suggests combining token usage with lines of code shipped and says some AI tools already provide data points around how much code was accepted. “Although ‘all the code was accepted’ is a very fuzzy measurement,” Jain admits.

That fuzziness is the state of things. The tooling to connect token spend to shipping outcomes doesn’t fully exist yet. The signals are noisy, and every framework requires tradeoffs between rigor and trust.