TLDR Tech / Dev
We are Changing our Developer Productivity Experiment Design
METR's first study in early 2025 on the impact of AI on developer productivity found that AI tools caused a 19% slowdown in task completion for experienced open-source developers. To re-evaluate AI's evolving impact, a new, larger experiment was launched in August 2025 using the latest AI tools. However, this second study yielded unreliable results due to selection bias, as many developers refused to participate without AI or chose only tasks where AI was highly beneficial. As a result, they are rethinking their methodology.
Read the full story