The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Not inevitably. Workplace studies have found higher task throughput and reported time savings, but a randomized trial involving experienced developers on familiar, mature projects found that AI access increased completion time. A separate, relatively small study found signs that heavily delegating a new coding task can come with weaker immediate understanding. The results differ because the people, work, tools, and outcomes differ; they do not show that AI coding tools universally help or harm engineers.
What the studies actually measured
“Productivity” can mean more tasks completed, less elapsed time on a task, or time developers say they saved. Those are not interchangeable measures. Nor does faster completion by itself establish better code: the cited studies do not provide a single, comparable measure of code quality across settings.
As an Amazon Associate I earn from qualifying purchases.
Skill claims need the same care. A quiz taken just after learning a library measures immediate comprehension, not whether someone will retain the knowledge, debug independently months later, or develop more slowly over a career. The evidence below addresses different parts of the question rather than delivering one universal score for AI-assisted engineering.
Recommended Free Tools
| Study and setting | Participants and design | Reported result |
|---|---|---|
| Microsoft Research, three company field experiments, summarized June 2025 | 4,867 developers across Microsoft, Accenture, and an anonymous Fortune 100 company; randomized access to an AI code-completion assistant in ordinary work | 26.08% more completed tasks in the combined analysis (standard error 10.3%). |
| METR, mature open-source projects, preprint submitted July 12 and revised July 25, 2025 | 16 experienced developers, 246 tasks, randomized comparison; developers averaged five years of prior experience in the projects | Tasks took 19% longer when early-2025 AI tools were allowed. |
| UK Government Digital Service, public-sector trial, November 2024 to February 2025 | 424 survey respondents across 31 departments, within a trial involving more than 50 public-sector organisations; survey responses paired with usage data | Respondents reported that 65% completed tasks faster and saved an average of 56 minutes per working day. |
| Anthropic, learning a new Python library, study page published in 2026 | A relatively small randomized study; participants used a self-guided task to learn Trio, with starter code and an AI assistant available to some | AI users finished faster on average, but the productivity improvement was not statistically significant. High-reliance interaction patterns were associated with lower immediate quiz scores. |
| Microsoft Research, “Dear Diary,” published in the 2025 ICSE-SEIP proceedings | Surveys, a randomized trial, and a three-week diary study at a large multinational software company | Perceived usefulness and enjoyment increased with sustained use; trust in AI-generated code did not change. |
Why workplace productivity findings differ
AI can raise throughput in some company settings
The three Microsoft Research field experiments tested access to an assistant that offered intelligent code completions during ordinary company work. The combined result concerns completed-task throughput in those participating settings; it does not show that every task took less time or that code quality improved. The researchers describe the individual experiments as noisy, and report that less experienced developers adopted the assistant more and saw larger gains. That pattern suggests experience and adoption may affect the result, but it does not mean every less experienced developer will benefit equally.
#1 Best Overall
Familiar, complex projects can impose a different cost
In METR’s randomized trial, developers worked in mature open-source projects they already knew well. They primarily used Cursor Pro and Claude 3.5 or 3.7 Sonnet. Before the trial, participants expected AI to reduce their time by 24%; afterward, they estimated it had reduced time by 20%, even though the measured result went the other way. The authors say experimental artifacts cannot be ruled out entirely, while noting that their robustness checks made them think experimental design was unlikely to be the main cause of the slowdown.
This is a meaningful counterexample, not a forecast for all engineering work. It applies to a small group of experienced developers, familiar projects, specific tasks, and early-2025 tools. It also illustrates why a developer’s impression that an assistant is saving time should not be treated as a measurement of task duration.
Public-sector time savings were self-reported
The Government Digital Service combined survey responses and tool-usage data; its main survey analysis included 33 job titles, and 73% of respondents said they had at least five years of coding experience. The report’s estimate of roughly 28 working days saved annually follows its stated calendar assumptions and extrapolates the reported daily average; it is not a directly observed annual saving for each developer. The report also notes uneven rollout and adoption, assumptions about how representative respondents were and how much they worked, the short trial period, and the absence of long-term measurement. These findings are useful evidence about users’ experience, but they are not a randomized comparison of task completion time.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Does AI use weaken coding skills?
Heavy delegation was linked to weaker immediate comprehension in one study
Anthropic’s study asked participants to learn Trio, a new Python library, in a self-guided coding task. They received starter code and a brief explanation; the AI assistant could access their code and generate a solution. The researchers assessed coding mastery, including debugging and code-reading abilities. Some participants used as much as 11 minutes—30% of their allotted time—to compose up to 15 AI queries, showing that interaction itself can consume a meaningful part of a learning task.
In a qualitative analysis, patterns involving high reliance—handing over the code, gradually delegating all writing, or relying on AI to debug—averaged below 40% on the immediate quiz. Participants who generated code and then checked their understanding, asked for explanations alongside generation, or asked conceptual questions and solved errors themselves averaged at least 65%. The study’s authors explicitly caution that these groupings do not establish that the interaction pattern caused the score: people who chose different patterns may also have differed in other ways.
An immediate quiz cannot establish lasting skill loss
The study was relatively small and tested understanding shortly after the task. It does not establish whether AI use causes long-term loss of retention, independent debugging ability, or skill growth. The other cited workplace studies do not answer those questions either. Longitudinal evidence about engineers’ independent capabilities over months or years is not established by these findings.
Rank #4
How to judge claims about AI coding tools
When someone says an AI assistant makes developers faster—or less capable—check what the evidence actually covers:
- Who participated? A result from experienced maintainers working in familiar projects may not transfer to junior engineers, a new codebase, or a different organisation.
- What was the task? Repetitive code completion, maintenance in a mature project, and learning an unfamiliar library place different demands on a developer.
- Which tools and when? Findings involving early-2025 tools do not automatically describe later tools or every assistant.
- What outcome was recorded? Completed-task counts, measured elapsed time, self-reported savings, and quiz scores answer different questions.
- Was the comparison randomized? Randomized access can support stronger causal comparisons than survey reports, although a randomized result still applies to its own participants and setting.
- Was learning tested over time? Immediate comprehension is not the same as later retention or independent problem-solving.
What engineers can take from this evidence
The studies do not prescribe a single workflow, but the learning results point to a practical distinction: using AI to support understanding is different from handing off the reasoning. For work where learning matters, an engineer can ask for a concept or explanation, inspect generated code, and try to explain or debug it independently rather than treating a working output as proof of understanding. For productivity evaluations, teams should measure the outcomes they care about—such as completed work, elapsed time, review effort, and quality—rather than treating reported time saved as a substitute for all of them.
Best Value
Microsoft Research’s separate “Dear Diary” study adds a reminder that usefulness and trust are distinct judgments: sustained use increased developers’ reported usefulness and enjoyment, while their views of the trustworthiness of AI-generated code stayed unchanged. Those perceptions and reported work-practice changes describe experience, not coding speed or skill retention.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
