Study finds AI tools made open source software developers 19 percent slower

Coders who used AI self-reported that they worked 20% faster, but an objective study found AI actually made coders 19% slower.

This study confirms that while AI can help for general low-level tasks, humans with deep knowledge of the specific project could spend far more time working on higher-level tasks, rather than hunting for errors in the work generated by AI.

I find that students who use AI to “save time” on the simpler scaffolding assignments simply aren’t equipped to do the higher-order thinking that’s required when I ask them to start connecting and building new ideas out of the early components. Yes, a bot can summarize a dry scholarly article, and with the right prompts bots can articulate opposing arguments and rebuttals, but students who copy-paste what the bots churn out don’t actually develop the critical thinking skills that empower them to do the important work I really want to assess.

AI tools tended to reduce the average time those developers spent actively coding, testing/debugging, or “reading/searching for information.” But those time savings were overwhelmed in the end by “time reviewing AI outputs, prompting AI systems, and waiting for AI generations,” as well as “idle/overhead time” where the screen recordings show no activity.

Overall, the developers in the study accepted less than 44 percent of the code generated by AI without modification. A majority of the developers reported needing to make changes to the code generated by their AI companion, and a total of 9 percent of the total task time in the “AI-assisted” portion of the study was taken up by this kind of review.

—Ars Technica