Stop trusting the hype: AI made expert coders 19% slower. They felt 20% faster.
AI made experienced coders 19% slower in a real timed test, while they believed they were 20% faster.
We didn’t run this. METR did. ModelSignal read the published work and checked every number below against it — the sources and the dates we read them are at the end.
16 expert coders · 246 tasks
19%
slower with AI tools.
Measured. Not felt.
METR, July 2025. Timed on their own code.
Felt vs measured
What they felt
20%
Faster, they said
Forecast before24% faster
The stopwatch
19%
Slower, with AI
Tasks timed246
Time it.
The coders were sure AI sped them up. The stopwatch said the opposite.
METR · 16 developers · July 2025
The proof
METR's July 2025 report and the 2026 rerun. Read 13 Oct 2026.
The full breakdown
Somebody else ran this experiment; we read it. Every number below is quoted from METR's own publications, with the date we read them.
The 2025 randomized trial (published 10 Jul 2025)
| Developers | 16, experienced, on repositories they had contributed to for years (averaging 22k+ stars, 1M+ lines of code) |
| Tasks | 246 real issues, each randomly assigned AI-allowed or AI-disallowed |
| Tools | primarily Cursor Pro with Claude 3.5/3.7 Sonnet |
| Result | with AI, tasks took 19% longer |
| Forecast beforehand | developers expected to be 24% faster |
| Belief afterwards | developers still believed they had been 20% faster |
The 2026 follow-up (published 24 Feb 2026)
| Data | 57 developers, 143 repositories, 800+ tasks, August 2025 to early 2026 |
| Returning developers | estimated 18% slower (confidence interval −38% to +9%) |
| Newly recruited developers | estimated 4% slower (confidence interval −15% to +9%) |
| METR's own reading | "our data is only very weak evidence"; a speedup is probable but not measured reliably, because developers increasingly declined to work without AI |
Sources
The catches
- The 2025 trial covers 16 experienced developers on code they know deeply, with early-2025 tools; METR says the historical result no longer reflects current models.
- The 2026 follow-up estimates are not statistically established; METR calls its data weak evidence and cites selection effects.
- Speed is the only outcome measured; code quality and enjoyment were not the primary outcome.
Every number, and where it’s quoted from
- 16 developers, 246 tasks“16 experienced developers from large open-source repositories … 246 total”metr.org 2025-07-10
- 19% more time with AI“When developers are allowed to use AI tools, they take 19% longer”metr.org 2025-07-10
- forecast 24% faster“expected AI to speed them up by 24%”metr.org 2025-07-10
- felt 20% faster afterwards“they still believed AI had sped them up by 20%”metr.org 2025-07-10
- Cursor Pro with Claude 3.5/3.7 Sonnet“primarily Cursor Pro with Claude 3.5/3.7 Sonnet”metr.org 2025-07-10
- repos 22k+ stars, 1M+ lines“averaging 22k+ stars and 1M+ lines of code”metr.org 2025-07-10
- 57 developers, 143 repos, 800+ tasks“57 developers, across 143 repos, and 800+ tasks”metr.org 2026-02-24
- returning developers 18% slower, CI −38% to +9%“-18% with a confidence interval between -38% and +9%”metr.org 2026-02-24
- new developers 4% slower, CI −15% to +9%“-4%, with a confidence interval between -15% and +9%”metr.org 2026-02-24
- weak evidence; speedup probable but unmeasured“our data is only very weak evidence for the size of this increase”metr.org 2026-02-24
- developers declined to work without AI“Developers increasingly declined participation rather than work without AI tools”metr.org 2026-02-24
