How to measure AI developer productivity (2026)

Your team rolled out Claude Code and Cursor three months ago. Pull requests are up, the licenses are in use and someone asks the obvious question: is it paying off? You search for an answer and find one experiment saying experienced developers took 4% to 18% less time, another saying 55.8% faster, a third saying about 21% less time and a GitHub-wide study where commits rose 240% while releases rose 30%. All of them are real, and none describes your team.

The disagreement has causes you can read off the study designs, and they tell you what to measure instead. This guide reconciles the main studies, lays out a metric model by layer and works one example end to end: human effort, review, rework, and AI cost across two tools.

How do you measure AI productivity in engineering?

AI productivity is measured as human effort and AI cost per accepted change, not as usage or code volume. The unit is the accepted change: a pull request, ticket or migration that met the criteria the team agreed on before starting. Counting accepted changes keeps abandoned sessions in the picture, because what they cost has to be paid by the changes that did get accepted. It also keeps the reviewer in the picture. A fix that takes ten minutes to write and forty to review is not a ten-minute fix.

What does the research say, and why do the studies disagree?

The studies disagree because they measure different tasks, with different people, against different baselines. Side by side, they stop contradicting each other.

Three variables explain most of the spread. The task: a greenfield server and a bug in a mature repository are different jobs. The baseline: Google and Copilot compared against engineers working without AI on a fresh task, while METR studied experienced developers in repositories they already knew. And the scope: time on a task, commits and shipped releases are three different measurements, and NBER found that a 240% rise in commits became a 30% rise in releases.

METR’s 2026 numbers also show the perception gap, although they come from two different samples. In its May survey, technical workers reported a median 3x speedup but only 1.4x to 2x more value, and METR warns that survey results are not necessarily grounded in reality. Its own February experiment measured 4% to 18% less time, with intervals that include zero. That is the best argument for putting system data next to surveys. Survey answers tell you how the work feels, and records tell you what happened to it.

Which metrics should you track, and in what order?

Track six layers in order, because each one explains the next. If adoption is low, a cost or delivery number says little. If delivery improves while stability falls, that is the pattern to investigate first.

The model borrows from two public frameworks. DORA’s ROI of AI-assisted software development links AI adoption to capabilities, then to the DORA metrics, then to productivity and experience, and only then to cost and revenue. DORA also says that company-wide aggregates are meaningless and individual scores are harmful. The SPACE paper makes the broader point that developer productivity “cannot be measured by a single metric or dimension”. Layer 5 is where the DORA guide applies, and the 2025 DORA findings show why it matters: AI adoption tracked with higher throughput and lower stability.

How do you calculate net effort per accepted change?

Net effort per accepted change is the human minutes spent preparing, reviewing and correcting a change, divided by the number of changes that met the acceptance criteria. Compare it with the same figure for comparable changes made without AI, and keep queue time out of it, so that waiting does not pass for effort.

Here is an illustrative example, with invented numbers chosen to show the arithmetic. A baseline batch of 30 comparable changes was done without AI. A second batch of 30 was done with AI: 18 with Claude Code and 12 with Cursor. Minutes include everyone who touched the change.

Preparation fell by 246 minutes. Review rose by 78 and correction by 48. The net reduction is 246 minus 126, which is 120 minutes, or 120 ÷ 900 = 13.3%. Reporting only the first row would claim a 39% cut in preparation time.

xychart-beta
  accTitle: Minutes per accepted change, baseline versus AI batch
  accDescr: Illustrative. Preparation 21 minutes in the baseline, 12.8 with AI. Review and verification 6 in the baseline, 8.6 with AI. Correction 3 in the baseline, 4.6 with AI.
  title "Minutes per accepted change (illustrative)"
  x-axis ["Preparation", "Review and verification", "Correction"]
  y-axis "Minutes" 0 --> 24
  bar [21, 6, 3]
  line [12.8, 8.6, 4.6]

The chart is the whole argument of this section in one picture: the saving sits in the first column, and part of it is paid back in the next two.

Both tools land on 26 minutes per change, but the shapes differ. The agentic Claude Code sessions spend 12 minutes on preparation and 9 on review; the editor-based Cursor work spends 14 and 8. These batches also contain different kinds of changes, so the table is not a ranking of tools. Comparing tools needs matched task types.

How do you calculate AI cost per accepted change?

AI cost per accepted change is the total AI spend attributed to a group of changes, including sessions that were abandoned, divided by the number of accepted changes in that group. Put every tool on this same denominator, because seats, tokens and credits are billed in different units.

Now price the minutes at an illustrative US$60 an hour, so US$1 a minute. The baseline batch costs US$900. The AI batch costs 780 in labor plus 138 in AI spend, US$918: a 2.0% increase, or US$30.60 against US$30.00 per change. Each change saved 4 minutes, worth US$4.00, and the AI behind it cost US$4.60.

So on cost alone, this workflow loses by a small margin. The case for it has to rest on something else: capacity released for other work, a shorter lead time, fewer defects or a better experience. Say which one, and measure it. Also note that salary-priced minutes are not cash savings unless the freed time goes somewhere.

What mistakes make AI productivity numbers misleading?

Five mistakes recur in AI productivity numbers.

• Reported hours times salary. Self-reports describe an experience. In METR’s 2026 survey, the median self-reported speedup was 3x while the median value gain was 1.4x to 2x. Use reported hours to decide where to look.
• Ranking individuals. Heavy users may take harder tasks, run several agents at once and review more. DORA suggests breaking usage down by team, application or use case, not by person. Tell people the data is not used for individual evaluation.
• Missing data as zero. A tool without telemetry shows up as zero usage, and the person who prefers it looks like a non-adopter. Flag the gap as a coverage gap.
• Task substitution. AI also makes people attempt work they would have skipped. METR’s May 2026 analysis shows that, under simplifying assumptions, uplift measured on old tasks is a lower bound and uplift on new tasks an upper bound on uplift in value. Their illustration uses +33% and +50%.
• Timing concurrent agents. METR found time measurements unreliable for developers running several agents at once. Count accepted changes and sampled human effort, not session duration.

PR counts deserve the same suspicion. In a study of 806 open-source projects that adopted Cursor, presented at MSR 2026, lines added rose 281% in the first month while commits rose 55%, so each commit carried far more code. DORA notes that this volume expands the review burden. “More PRs” and “more output” are different claims.

How do you run a first 30-day evaluation?

A 30-day evaluation follows one workflow, one acceptance definition and one set of sources, and ends with a decision: expand, change the workflow or keep observing. It is a first read. DORA expects a productivity dip when teams adopt AI and says its depth and duration remain unpredictable.

flowchart TD
  accTitle: The 30-day AI productivity evaluation
  accDescr: Days 1 to 5 set the baseline and acceptance definition. Days 6 to 10 connect sources and label coverage gaps. Days 11 to 25 record effort on a sample and run weekly outlier reviews. Days 26 to 30 check follow-up failures and compute the results. The evaluation ends in expand, change the workflow, or keep observing.
  A[Days 1 to 5: baseline and acceptance definition] --> B[Days 6 to 10: sources connected, gaps labeled]
  B --> C[Days 11 to 25: sampled effort, weekly outlier review, survey on day 15]
  C --> D[Days 26 to 30: follow-up window, net effort, cost, lead time, stability]
  D --> E{Decision}
  E --> F[Expand to more teams]
  E --> G[Change the workflow and rerun]
  E --> H[Keep observing another period]

Aim for at least 20 comparable changes per group. That is a planning number, not a statistical threshold. With fewer, report the records and skip the percentage.

Frequently asked questions about measuring AI productivity

Does AI make developers slower?

In METR’s February 2026 experiment, returning developers took 18% less time and new ones 4% less, with intervals that include zero, and METR calls its data very weak evidence. Its May 2026 survey found technical workers self-reporting a median 3x speedup but 1.4x to 2x more value. The effect depends on the task, the person and the tool, and you have to measure it locally.

Is acceptance rate a good metric?

It belongs to the adoption layer. Acceptance shows how often a suggestion was kept, not whether the change survived review or production, and it cannot see what the reviewer paid.

How long before AI impact shows?

DORA expects an initial productivity dip and says its depth and duration remain unpredictable, so there is no fixed wait. Use 30 days to find out whether you can measure the workflow at all.

Can you measure individual developers?

You can, but you should not use it to evaluate them. Differences in task difficulty, review load and agent use swamp the signal, and DORA advises looking by team instead.

How do you measure agents running in parallel?

Count accepted changes, the cost of every session behind them and sampled human minutes for preparation and review. Elapsed session time overstates effort when several agents run at once.

How do you calculate AI ROI without inflating it?

Use net effort including review and correction, add all AI spend including abandoned sessions, and compare against comparable work without AI. Then require a named benefit: capacity, lead time or quality.

What should you do next?

Pick one change type your team repeats, such as bug fixes or dependency upgrades. Record preparation, review and correction minutes for twenty of them, and the AI cost behind each. Then check what happened to those changes in production with the DORA metrics guide, because the stability layer is where the 2025 research found the cost.