Your engineering org just rolled out Copilot to 400 developers. Three months in, someone produces a dashboard. Pull request cycle time is down 28%. Lines of code per developer are up. Acceptance rates on AI suggestions look healthy. Leadership is pleased. The next budget cycle funds a broader rollout.

The problem is that nobody can tell you whether the software got better, whether the bugs that shipped last quarter were a new kind of bug, or how much time senior engineers now spend reviewing AI-generated code they didn't write and don't fully trust. The metrics said "faster." The codebase said something more complicated.

I've sat in enough quarterly reviews to recognize this pattern. The numbers go up, the narrative is positive, and the uncomfortable questions don't get asked because there's no instrument pointed at them. I'm not arguing that DORA, SPACE, or DevEx frameworks are broken. Neither was designed to isolate the contribution of a single tool or practice, and teams that use them for AI-specific attribution are misapplying the instrument. What most teams haven't configured is a complementary layer of metrics built for a world where AI produces the first draft and humans verify the result.

AI didn't just speed up existing tasks. It changed what a task is.

Before Copilot, a developer's core loop went like this. Understand the requirement, design the approach, write the code, test it, ship it. The cognitive work was in the writing. After Copilot, the loop shifted. Understand the requirement, prompt or steer the generation, read what came back, evaluate whether it's correct, decide what to keep, revise what isn't right, test the revised version. The cognitive work moved from generation to verification and judgment.

That's a different job wearing the old job's clothes. I watched a legal team adopt AI contract review and the senior partners didn't get faster at reading redlines. They got faster at deciding whether to trust the machine's redlines, which is a completely different skill. The cognitive shift from producing to verifying is the same across every function that adopted AI tooling.

The impact isn't uniform either. AI creates a bimodal distribution across engineering teams. Mid-level engineers with solid fundamentals get enormous leverage because they can evaluate output effectively. Juniors can't reliably judge whether AI-generated code is correct. Experts can often write better code faster than they can steer the AI toward the right solution. Any measurement framework that treats "engineers" as a monolithic group will average out the most interesting signal.

When Microsoft published its early research on GitHub Copilot productivity, the headline number was that developers completed tasks 55% faster. What got less attention is that the study measured task completion time on a bounded, well-defined coding task. It didn't measure code quality, long-term maintainability, review burden on teammates, or whether the "faster" code introduced subtle issues that surfaced later. The metric was real but narrow. And narrow metrics, presented without context, become organizational blind spots.

Five metrics I see in nearly every AI productivity report, and what each one actually tells you.

Lines of code and PR throughput. Measures volume. With AI, more output might signal the opposite of productivity. If your team's PR count doubled but your test coverage didn't move and your average review depth dropped, you've increased your maintenance surface without increasing your confidence in it.

Self-reported time saved. The most common vendor metric, and the least reliable. In the teams I've observed, developers asked "how much time did AI save you?" overestimate by 1.5-2x compared to actual commit timestamps and cycle times. It's anchoring bias compounded by demand characteristics and social desirability pull toward positive results.

DORA metrics in AI-augmented teams. Deployment frequency and lead time might improve, and the improvement might have nothing to do with AI. If your team also adopted trunk-based development or improved their CI pipeline in the same period, you can't credit AI for the delta.

Cost per automated action. Usually excludes verification overhead. If every automated action requires human review, you've automated the draft, not the action.

Adoption rate and seat utilization. "87% of our developers used Copilot this month." That tells you procurement was successful. I've seen teams with 90%+ adoption where the most experienced engineers quietly turned off suggestions.

The pattern across all five is the same. Every popular metric measures something that existed before AI. None capture the new costs and failure modes AI introduced.

I suggest start instrumenting these four metrics to get deeper insights.

Effective first-pass acceptance rate. Vendor dashboards report "acceptance rate," the percentage of AI suggestions a developer clicks "accept" on. But acceptance isn't approval. If your team accepts 80% of suggestions but meaningfully modifies 40% of those before the PR merges, the effective first-pass rate (what actually ships without substantial human correction) is closer to 48%.

What counts as modification is meaningful logic changes, structural rewrites, or replacements that alter the behavior of the suggestion. Operationally, compare the accepted suggestion snapshot against the final committed diff. If the Levenshtein distance exceeds 30% character change, flag it as a rework event. That threshold is crude (renaming variables might exceed it while a single critical logic fix might not), so treat it as a triage filter and layer in AST-level diffing as your tooling matures.

You'd need to correlate IDE telemetry with VCS commits to get this kind of information. Prototyping for a single team is a sprint-or-two exercise. Making it production-grade for 400 developers is two to three quarters of platform engineering. And if you're subject to GDPR or have a works council, capturing editor-state telemetry triggers a Data Protection Impact Assessment and likely requires a collective agreement. That alone can add 6-12 months.

Decision threshold: Above 60%, AI is generating genuinely useful output. Between 40-60%, the tool provides a starting point but your team is doing significant rework. Below 40%, it's functioning as an expensive autocomplete.

Verification overhead delta. Time spent reviewing AI output is a first-class cost that most organizations don't track separately. Reviewing code you wrote is a recall exercise. Reviewing code an AI wrote is a comprehension exercise and the latter often takes more effort.

Measure this through your code review platform's timestamp data. Calculate median wall-clock time from "reviewer assigned" to "approval," segmented by AI-assisted versus human-authored PRs. If AI-assisted PRs consistently take 25%+ longer to review, the verification cost is eating your speed gains.

Also track reviewer queue depth and approval-without-comment rate. If your team's PR output increased 40% after AI adoption but your reviewer pool didn't grow, you have a bottleneck that speed metrics will never surface. Once queue depth crosses 5 open PRs and approval-without-comment pushes above 30%, you've saturated your review pool.

Error class shift. Before AI, your bugs had familiar shapes. Off-by-one errors, null pointer exceptions, missed edge cases. With AI-generated code, I've seen a qualitatively different pattern emerge. Code that's syntactically valid, passes type checks, even passes shallow unit tests, but embodies an incorrect understanding of the business domain. It looks more polished than the typical junior-engineer mistake, which is exactly what makes it dangerous.

One example from a PR I reviewed. Agent generated a currency conversion function using banker's rounding (round-half-even). Perfectly reasonable, except the system's financial contracts specified round-half-up. The difference was $0.01-$0.03 per transaction, only visible in monthly reconciliation. The code compiled, passed all existing tests, and looked more professional than what most humans would've written.

Developers make sloppy mistakes that look sloppy. AI makes confident mistakes that look polished.

Track this by adding an "AI-origin" tag to your defect taxonomy. Over time, you'll build a distribution that shows whether AI shifts the types of bugs, not just the rate. If AI-origin defects consistently represent more than 20% of your post-deployment bugs while AI-assisted code represents less than 20% of your codebase, the defect rate is disproportionate.

Test quality on AI-generated code. When the same model writes both the implementation and the test assertions, you get circular validation. The tests verify that the code does what the code does, not that it does what the specification requires. I've seen this produce 90%+ test coverage numbers that catch zero domain-logic errors. When AI generates both code and tests, require that a human writes or independently reviews the test assertions.

A note on security. AI-generated code also introduces distinct vulnerability patterns (insecure defaults, improper input validation, hardcoded credentials that look like placeholders) that correctness testing alone won't catch. If your codebase is security-sensitive, segment your static analysis findings by AI-assisted versus human-authored code the same way you'd segment defects.

One caveat across all four metrics: use them for team-level learning and portfolio decisions, not individual developer performance reviews. The moment any of these becomes a performance target, developers will optimize the metric rather than the outcome it measures.

Everything I've proposed costs engineering effort. The right level depends on your scale.

Under 50 engineers. Don't build custom instrumentation. Pair your existing speed metrics with quality metrics (revert rate, incident rate), segment dashboards by AI-assisted versus not, and have honest quarterly conversations about what's working.

50-200 engineers. Assign one platform engineer for a quarter to build review timestamp analysis and basic rework tracking. Budget $80-$100K in engineering time, assuming you already have a code review platform with API access and a basic data pipeline. Without those prerequisites, double it. Skip the full IDE-to-VCS telemetry pipeline unless you have a mature platform team.

200-1000 engineers. The full measurement stack pays for itself. Allocate budget for one to two platform engineers over two to three quarters. That's 12-25% of your AI tooling spend to actually know whether the other 75-88% is working.

1000+ engineers. You need a dedicated measurement function. Treat AI measurement the same way you treat infrastructure observability. You wouldn't deploy a $2M infrastructure platform without an SRE team monitoring it.

A measurement system that can only confirm value is just marketing. You need to define, in advance, what results would make you reduce or discontinue AI tooling for a given team or use case.

If, after two quarters, a team shows effective first-pass acceptance rate below 35%, AI-assisted defect rate more than 1.5x the baseline, review overhead delta above 30%, and no improvement in downstream outcome quality, that team isn't getting value from the tool regardless of what the adoption dashboard says. The correct response is to investigate why and either address the root cause or reallocate the seats.

The goal is ensuring your measurement system can say "no" as clearly as it can say "yes." If every possible result leads to "keep investing," you have a rationalization system, not a measurement system.

If you're an engineering leader who just approved a six-figure AI tooling budget, ask yourself one question. Can you tell, with specifics, what your team's effective first-pass acceptance rate is? Not the vendor acceptance rate or the self-reported time savings. It should be the rate at which AI-generated output actually ships without meaningful human correction.

If you can't answer that, you're measuring enthusiasm rather than AI productivity. And enthusiasm is how organizations end up doubling down on tools that make dashboards look good while the actual work gets quietly harder in ways nobody instrumented.

DORA, SPACE, and DevEx are solid frameworks. They just weren't configured for a world where AI produces the first draft and humans verify the result. Adding verification costs, rework loops, error class shifts, and outcome quality to your existing measurement stack closes that gap.