Recommended Free Tools
A 2026 Google Research study reports that using a written contract to guide AI-generated tests improved bug detection and branch coverage in its evaluation. It does not test the central extra step proposed by Gal Arav: keeping independently written acceptance criteria from the agent that implements the code. That distinction matters. Evidence that contracts can help generate tests is not evidence that hidden, independent criteria catch more real defects.
What the Google study measured
In “Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation,” the agents use a two-stage process. First, an agent reasons about existing code and documents its preconditions, postconditions, and undefined behavior. Then that semi-formal contract guides test generation.
As an Amazon Associate I earn from qualifying purchases.
Against a traditional test-generation baseline, the paper’s abstract reports these results for its evaluation:
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minute- Bug detection: 9.8 percentage points higher with the contract-guided approach (p = 0.0352).
- Branch coverage: 2.5 percentage points higher (p = 0.0034).
- Judge preference: an LLM-as-a-Judge preferred the generated suites in 77.8% of pairwise comparisons with baseline suites and 56.7% of comparisons with human-authored suites.
The last figures describe a judge’s pairwise preferences, not a universal or objective ranking of test quality. Likewise, the bug-detection and coverage differences are results in the paper’s stated evaluation setting, not promised gains for other codebases or deployments. Branch coverage indicates which branches tests exercise; it does not by itself show that the tests detect every important defect or that the software is correct.
What the study did not test
The contract in the Google study is derived by analyzing code that has already been written. The paper therefore evaluates whether a documented, implementation-informed contract can help an agent generate tests. It does not establish that the contract is independent of the implementation, or that withholding acceptance criteria from the coding agent improves defect detection.
That is the gap in the argument signaled by the title: one half has a published comparative result, while the other remains a hypothesis. The study supports the narrower claim that contract-guided generation can improve measured test outcomes in its evaluation. It cannot be used as evidence for a different workflow merely because both approaches use contracts.
What Independent Spec-Driven Development proposes
In an October 6, 2026 article, Gal Arav describes Independent Spec-Driven Development (ISDD) as a separation of knowledge. The implementation agent receives requirements but not the acceptance criteria that will be used to judge its work. A separate test-generating agent or process applies those criteria. The proposed benefit is that independently authored criteria might expose implementation choices or defects that tests generated with the implementation agent’s shared context could miss.
Arav presents this as an open empirical question, not a proven advantage. His article reports limited personal experiments: attempts to see whether refining criteria improved tests did not show a clear effect in matched comparisons. Across 20 runs on two tasks, separate implementation and test agents reportedly disagreed in every run; examples involved rounding and an unresolved boundary choice. Those disagreements illustrate how agents can interpret a task differently. They do not show that the resulting tests found real bugs, or that the approach outperforms ordinary testing.
How the approaches differ
| Dimension | Google study | Arav’s ISDD proposal |
|---|---|---|
| Contract provenance | Derived from analysis of existing implementation code. | Acceptance criteria are independently authored rather than derived from the implementation agent’s code. |
| Agent access | The contract guides test generation; the study’s abstract describes no withheld-criteria design. | The implementation agent is not given the acceptance criteria used to evaluate its work. |
| Reported evaluation | Published abstract reports bug-detection, branch-coverage, and judge-preference results in its evaluation. | Article reports limited experiments and disagreements; it does not demonstrate improved real-bug detection over ordinary suites. |
| Evidence status | Comparative study results reported by Google Research authors in 2026. | Proposal and author-reported demonstrations in an October 6, 2026 article. |
The comparison is not a head-to-head test: the approaches differ in how the contract is created and who can see it, and the evidence comes from different kinds of evaluation. A measured improvement in one cannot settle the other.
The practical question is whether independent criteria find more real defects
The useful comparison is not simply whether two agents disagree, or whether a test suite covers more branches. It is whether tests produced from independently authored, withheld acceptance criteria detect more real defects than conventional suites or tests generated with shared implementation context. The results would need to distinguish defect detection from proxy measures such as coverage and subjective judge preference.
Rank #4
Arav’s article frames this as the unresolved question: “Whether that division catches more real defects is the open question, and it is what qikly is built to test.” The article asks readers to identify work that measures it; the quotation is a statement of the research goal, not a finding.
Acceptance criteria still need reachable fixtures
A criterion cannot expose behavior the test data cannot reach. If the available fixtures contain no values at a relevant boundary, a test generator cannot exercise that boundary simply because the criterion describes it. Arav’s article says qikly added --propose-fixtures to identify criteria that the available data cannot reach. That is a tool feature reported by the article, not independent evidence that the feature or the broader ISDD method improves defect detection.
Best Value
For teams considering the idea, the practical implication is to check both sides of a test: whether the acceptance criterion is independent of implementation choices, and whether the fixture can actually drive execution through the behavior it describes. Otherwise a test can be well specified yet unable to evaluate the intended case.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
