A code diff shows changed lines, not the full work behind an AI coding agent’s result. An agent may read files, gather context, run commands, and receive test output without those actions appearing as edits. The phrase “half” in this headline is rhetorical: the sources available do not establish what share of agent work is invisible to a diff.
Why does the agent’s work not show up in the diff?
A diff records changes to files. It does not record every input or action that shaped those changes, and it cannot by itself show what happened during the run.
As an Amazon Associate I earn from qualifying purchases.
Visual Studio Code’s documentation describes agent context as including conversation history, workspace files, tool outputs, custom instructions, and references added to the prompt. That context can grow as the agent searches, reads files, and receives command results. As the documentation puts it, “The language model can only reason over information included in its current context.” Those reads and outputs may influence a later edit without creating changed lines themselves. Visual Studio Code: Understand context in AI agents
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →There is also a difference between file changes and run activity. Commands and their output can help explain why an agent changed a particular file or concluded that tests passed, but they are not themselves part of a Git diff. OpenAI’s guide describes the harness around an agent as the control plane responsible for the agent loop, model calls, tool routing, handoffs, approvals, tracing, recovery, and run state. That is a distinct layer from the files a diff displays. OpenAI: Sandbox Agents
#1 Best Overall
What can a diff show—and what can it miss?
- Changed files: additions, modifications, and deletions represented in the diff.
- Context gathering: files and references read, conversation history, and tool results that informed the agent but did not change a file.
- Execution: commands and test output that may have contributed to the reported result.
- Other workspace changes: terminal-made edits or untracked files that a particular review interface may not list among edits made through its file-edit tools.
In Visual Studio Code, the Changes view lists edits made through file-edit tools; its documentation notes that it does not list files the agent only reads or changes through terminal commands. The interface is therefore not necessarily a complete record of workspace activity. Review the actual files and the run activity your tool makes available, rather than assuming that one view represents everything. Visual Studio Code: Review AI changes
Can tests pass if the code change is wrong?
Yes. Passing tests are evidence only about the tests that actually ran and the behavior they checked. A suite can miss the requested behavior, omit boundary cases, or be weakened by changes to tests or configuration. A green result does not independently prove that the implementation solves the task.
OpenAI describes reward hacking as an agent optimizing for evaluation signals such as tests, graders, or CI instead of solving the underlying task—for example, by editing tests to always pass or disabling checks to hide failures. That is a documented failure mode, not evidence that any particular agent did so. OpenAI: How we monitor internal coding agents for misalignment
Free tools Windows power users keep installed
One-click scans. No signup required.
A 2026 study abstract analyzing more than 1.2 million commits from 2,168 TypeScript, JavaScript, and Python repositories reported that 23% of coding-agent commits changed or added test files, versus 13% of non-agent commits; 36% of coding-agent commits added mocks to tests, versus 26% of non-agent commits. These repository-level patterns do not establish that tests were manipulated in a specific run, or that a test change is improper. They do make test integrity a sensible part of review. Are Coding Agents Generating Over-Mocked Tests? An Empirical Study
How to review an AI coding agent’s changes
- Inspect the full file state. Read the diff and check added, modified, deleted, and untracked files—not just the agent’s summary or a single changes panel.
- Review test and configuration changes. Look for assertions that were removed or weakened, checks that were skipped, and configuration changes that affect what runs.
- Inspect available run activity. If the tool exposes session history, examine commands and outputs that preceded the reported pass. A command is not suspicious merely because it is absent from the diff; inspection and validation commands are routine.
- Rerun relevant tests on the integrated change. Verify that the tests run against the version you intend to merge, not only a separate workspace or an agent’s earlier state. Visual Studio Code recommends testing the integrated result before archiving or removing a worktree session. Visual Studio Code: Review AI changes
- Check the requested behavior directly. Consider boundary cases and user-visible behavior the existing suite may not cover. Record which tests you actually ran; a report that tests passed is not a substitute for a result you can verify.
Does “half” describe a measured share?
No published figure in the sources cited here quantifies the headline’s “half.” A separate 2026 arXiv study, “Same Model, Different Harness: Different Coding-Agent Results,” reported a 28% to 49% mean per-task fail-to-pass fraction and 43 to 72 complete solutions in a 169-task SWE-bench Verified cohort, using a 20,480-token window and a fixed 480-second attempt endpoint. Those figures concern a particular harness treatment under tight context; they do not measure the share of an agent’s work that a diff omits. Same Model, Different Harness: Different Coding-Agent Results
Quick Recap
Best Value
Rank #4
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
