AI coding tools can speed up a defined coding task, but generating code is only one part of software development. Developers also have to understand unfamiliar systems, choose the right change, test it, review it with others and maintain it later. The gap is not that AI tools never help; it is that a local gain in code production does not automatically improve the whole workflow.
Why faster code generation is not the same as faster development
A coding assistant can suggest a function or help complete a bounded task. The surrounding work is less bounded: finding where a change belongs, understanding existing behavior, weighing trade-offs, checking edge cases, coordinating review and supporting the result after release. Those activities determine whether a suggestion is useful, safe and maintainable.
As an Amazon Associate I earn from qualifying purchases.
That distinction matters when judging productivity. More typing, more accepted suggestions or less time on one task may indicate a benefit, but none alone establishes that a team shipped better software sooner or reduced the work needed to sustain it.
What the studies measure—and what they do not
Long-term IDE activity: useful signals, not a productivity score
JetBrains Research analyzed two years of anonymized IDE telemetry from 800 developers—400 AI users and 400 non-users—covering 151,904,543 logged events, and paired it with a survey of 62 developers. AI users’ typing rose by nearly 600 characters per month, compared with 75 among non-users. More than 80% of survey respondents said their productivity had increased. These are different kinds of evidence: typed characters are an activity proxy, while the survey captures people’s perceptions. Neither measures shipped value directly.
The telemetry also showed about 100 more monthly deletions among AI users, compared with about 7 among non-users, and about six more monthly IDE activations among AI users while activations fell by about seven among non-users. These patterns can prompt questions about editing and switching back to the IDE; they do not prove AI caused extra rework or fragmented attention. Many survey respondents did not perceive matching changes in editing or context switching. The authors explicitly describe the telemetry as pattern-finding, not causal evidence. JetBrains Research’s account of the study
Controlled tasks: evidence for specific kinds of help
GitHub reported that Copilot users completed one controlled coding task 55% faster in its 2022 study. That result supports a narrower claim—that assistance can help on that defined task—not a general promise of a 55% productivity increase across software work. GitHub’s 2022 study
In a separate GitHub Customer Research study, experienced Python developers built API endpoints for a fictional restaurant-review web server. Among 202 valid submissions, developers using Copilot were 53.2% more likely to pass all 10 unit tests. The constrained task and test-based outcome make this evidence about performance in that exercise, not about long-term maintenance or team delivery. GitHub’s code-quality study
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsSurveys: reported adoption is not observed frequency
In a 2024 GitHub survey, updated in 2025, more than 97% of 2,000 enterprise respondents across the United States, Brazil, Germany and India said they had used AI coding tools at work at some point. This describes those respondents and the survey wording; it does not show how often developers use the tools or how much value they get from them. GitHub’s survey account
Rank #3
Maintenance and experienced reviewers
A 2025 preprint by Xu, Medappa, Tunc, Vroegindeweij and Fransoo analyzed open-source projects after Copilot adoption. The authors report 6.5% more code reviewed and 19% lower original-code productivity among experienced core developers in that setting. These findings raise a different question from task-speed studies: whether assistance shifts work toward review and maintenance, particularly for people who know a project deeply. They should not be treated as estimates for every company or engineering team. The authors warn that “productivity gains of AI may mask the growing burden of maintenance on a shrinking pool of experts.” The authors’ 2025 preprint
Why repository context, review and team practice still matter
Even when a tool can use surrounding code, a suggestion still has to fit the project’s conventions, architecture and goals. A developer must decide whether the change addresses the actual problem, whether tests cover likely failures and whether reviewers can understand and support the result. This is an explanation of the workflow gap, not a finding that every tool lacks repository context or that every suggestion requires the same amount of review.
Results can also diverge between perception and measured performance. TIME’s 2025 account of a METR study described 16 experienced developers working on complex projects: measured completion was about 20% slower even though developers estimated they were about 20% faster. The small, specific sample cannot establish how all developers fare, but it illustrates why confidence or satisfaction should be considered alongside observed outcomes. TIME’s report on the METR study
Recommended Free Tools
How to compare claims about AI coding productivity
Before treating two results as comparable, check what each study actually asked developers to do and how it counted success. A controlled task, a survey, IDE telemetry and an analysis of open-source maintenance answer different questions.
Best Value
- Task and realism: Was the work a short, defined exercise or a change in a mature project?
- Project familiarity: Did participants know the codebase, and could the tool use relevant repository context?
- Developer experience: Were participants beginners, experienced specialists or a mixture?
- Outcome: Did the study measure elapsed time, tests passed, code volume, review work, maintenance, or self-reported productivity?
- Method and sample: Was it a controlled task, survey, observational analysis or preprint, and how many people or projects were included?
- Setting: Which tools, versions, work practices and countries were represented?
These distinctions help explain why a real gain on a narrow task can coexist with more review work or no improvement in end-to-end delivery.
What teams should measure instead of counting suggestions
Teams evaluating an assistant should choose measures that reflect their own work rather than assume that more generated code means greater productivity. Pair operational data with developers’ accounts of where the tool helps or adds friction. GitHub recommends tailoring success metrics to organizational processes and goals; JetBrains’ authors caution against reading causal conclusions into their telemetry. GitHub’s discussion of productivity measures · JetBrains Research’s study limitations
- Track time and completion alongside quality indicators such as tests and defects.
- Include review effort and follow-up maintenance, not just the initial coding step.
- Compare similar tasks and experience levels, and record whether developers know the codebase.
- Ask developers where assistance changed their workflow, then check those reports against observed outcomes.
That approach can reveal whether a tool removes tedious work, moves work elsewhere or improves delivery for a particular kind of task. The available findings do not establish one universal effect for all developers, projects or tool versions.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
