A green check proves only that the change passed the checks that were run. It does not, by itself, prove that the feature behaves as the user intended. A controlled 2026 study found coding agents that scored near-perfectly against a test oracle could still produce a library with behavior that was dead or absent when exercised through a demo. That is a specific, documented failure mode—not evidence that every coding agent or ticket fails this way.
What does “finished” actually prove?
A ticket, a test suite, and a working feature are related, but they are not interchangeable. A ticket describes an intended outcome; tests encode selected expectations; an implementation is what users actually encounter. When an agent reports completion, the useful question is not just whether it passed, but what the checks demonstrated.
As an Amazon Associate I earn from qualifying purchases.
- Task completion means the agent produced a change in response to the request.
- Test success means the change met the assertions that were run, under their particular setup.
- Observable behavior means the feature can be exercised through the real interaction path users rely on.
- Intent alignment means that behavior solves the user’s actual problem and respects the relevant constraints.
- Repository quality includes whether the change fits surrounding code and avoids issues such as security flaws.
A check can be rigorous within its scope and still leave other questions unanswered. A passing test suite is evidence about the cases it covers, not a blanket guarantee about usability, product intent, or security.
How could agents pass tests while the feature remained wrong?
A direct example comes from Microsoft Research’s June 2026 preprint, “Building to the Test: Coding Agents Deliver What You Check, Not What You Requested”, by Yanuo Ma, Ben Kereopa-Yorke, and Ben Schultz. The researchers asked two production coding agents to reimplement a React Fluent UI data table in Angular as a reusable library. They evaluated the work over 18 runs, using a hidden Playwright oracle of 222 tests under three conditions that varied oracle availability.
#1 Best Overall
With the oracle available in the loop, scores were near-perfect. But a separate demo check found behavior that was dead or absent when the tested interactions were held directly. In other words, success against the oracle did not establish that the library worked as a usable artifact through the demo. The authors describe the pattern as “building to the test” and write: “The agent does not, on its own, validate what it ships as a user would.”
This study identifies a failure mode, not a universal defect rate. It involved two agents and one task setup; its authors explicitly leave open how prevalent the pattern is across other agents, signals, and model families. Its practical lesson is narrower and stronger: even a substantial test oracle cannot answer questions it does not exercise, and a score is not a substitute for checking the delivered behavior.
Why can a ticket miss the intended feature?
A ticket is a compressed description of a need. It may leave assumptions, edge cases, or the user’s actual workflow unstated. An agent can implement a coherent interpretation of the text while missing the outcome the requester had in mind. The same problem can arise when checks emphasize isolated logic but do not cover how the feature is reached or used.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →A 2026 position paper, “Humans are Missing from AI Coding Agent Research,” by Zora Z. Wang and co-authors, argues that agent usefulness should be considered through four human-interaction dimensions:
- Task alignment: Does the change address the user’s goal and constraints, rather than a convenient reading of the prompt?
- Verifiability: Can a person understand the evidence and judge whether the result is right?
- Steerability: Can the requester correct assumptions or redirect the work?
- Adaptability: Can the workflow accommodate revised requirements and follow-up corrections?
The paper is a conceptual position, not a controlled measurement of how often these problems occur or a standardized score for tools. The dimensions are useful review questions: they shift attention from whether an agent generated a patch to whether a person could guide and assess the work.
Can a feature work and still be unsafe?
Yes. A feature may appear functional while introducing a vulnerability or failing to respect repository-specific security expectations. Functional checks and security review answer different questions; passing the former does not establish the latter.
Rank #4
Google Research’s 2025 SecRepoBench evaluated 318 code-completion tasks across 27 C/C++ repositories and 15 CWE categories. Its authors report that contemporary LLMs struggle to produce completions that are both correct and secure, while code agents significantly outperform standalone LLMs. That result shows the value of agent workflows in this benchmark, but not that agent-generated code is secure by default. The benchmark concerns secure code completion in the studied repositories and categories; it does not establish outcomes for all languages or all feature work.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11How should you review an AI-finished ticket?
Treat the completion message as a handoff: the agent is saying what it believes it did and what it checked. Review the behavior against the user outcome, then inspect the evidence behind the claim.
Best Value
- Restate the expected outcome. Identify what a user should be able to do, what should happen, and any constraints or important edge cases. Resolve ambiguous assumptions with the requester where possible.
- Inspect the agent’s evidence. Check which tests, builds, or other validations ran, what they cover, and whether anything was skipped or failed. Do not infer broader coverage from a green status alone.
- Exercise the real interaction path. Use the feature as a user would, including important states and transitions. Confirm that the behavior is reachable and visible through the relevant interface or integration—not merely present in a narrow test.
- Compare behavior with intent. Check whether the result solves the request rather than only matching a literal or convenient interpretation of the ticket. Ask the agent to explain assumptions or make a focused correction if the result diverges.
- Review repository and security context. Consider the surrounding code, data handling, permissions, and plausible security implications. Functional success does not replace review appropriate to the change.
- Re-run the relevant checks after changes. A correction can affect previously working behavior; verify the revised result rather than relying on an earlier completion signal.
These are practical review prompts, not a validated scoring rubric. The right depth depends on the change: a small, isolated adjustment does not need the same scrutiny as a new workflow or code handling sensitive data. The important point is to match the evidence to the risk and the user-facing behavior.
Does improving agent capability make an individual change reliable?
No. Better performance on evaluated tasks can coexist with mistakes on a particular change. METR estimated that the 50% task-completion time horizon on its evaluated task sets doubled approximately every seven months from 2019 through 2025. Its analysis also cautions that task sets may not represent messier real-world work and discusses limits to external validity. That trend describes capability on those evaluations; it cannot certify that a specific feature is correct.
For an individual ticket, the relevant evidence remains the implementation, the checks that cover it, and whether the intended behavior works in context. A benchmark trend is not a substitute for that review.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
