AI-assisted test automation is more credible when the agent that writes the code cannot see the acceptance criteria used to test it. In Gal Arav’s workflow, one agent implements written requirements while a separate agent turns hidden acceptance criteria into tests. The arrangement can make a failure a useful challenge to the implementation—but it does not prove the criteria themselves are correct.
How the spec-driven workflow separates coding from testing
The method applies separation of duties to software verification: give the coding agent the requirements, but withhold the acceptance criteria. A different testing agent receives those criteria and writes tests without seeing the implementation. As Arav puts it, “the person who builds the system must never be the person who verifies it.”
As an Amazon Associate I earn from qualifying purchases.
The important safeguard is the information boundary, not simply assigning two labels to one agent or asking a single agent to be impartial. If the implementation agent can inspect the criteria, the test-writing process is no longer independent in the intended sense.
What the separation can establish
If a test derived from criteria the coding agent did not see fails, it identifies a disagreement between the implementation and a standard the coder was not given. It is evidence that the implementation may not meet that standard; it is not by itself proof that the standard describes the right behavior.
What a passing test can establish
A pass shows that the code satisfied the exercised tests. Its evidential weight depends on the history of the code and criteria: if code was written during the workflow by an agent that could not see the criteria, a pass is stronger evidence against tailoring to those criteria. It still does not establish correctness beyond the specified and exercised behavior.
New code and existing code provide different evidence
The distinction is whether the person or agent responsible for the implementation could have seen the acceptance criteria—not merely whether tests were created separately.
| Context | What a failing test indicates | How much a passing test supports |
|---|---|---|
| Code written during the workflow, with criteria withheld from the coding agent | A possible failure to meet criteria that the coding agent did not see. | Stronger evidence against implementation tailored to those criteria, though only for tested behavior. |
| Pre-existing code | A real finding if the tests were generated from criteria without reading the implementation. | Weaker evidence: the code’s author may have seen the criteria. Commit order can offer a limited clue, but commit dates do not prove when code was written or what its author saw. |
In both cases, test quality depends on whether the test data can actually trigger the criterion. A rule that no sample can exercise contributes little evidence, even if the suite reports a pass.
Why the exact boundary belongs in the requirement
Arav’s example concerns radar samples used to calculate time headway and warn when it falls below a two-second threshold. A coding agent initially accepted a zero-metre gap. A separate test, derived from criteria hidden from that agent, caught the row; the agent then changed its lower-bound check in response.
The example also exposed an ambiguity: does “breaks the two-second rule” mean strictly below two seconds, or at or below two seconds? If that decision exists only in hidden acceptance criteria, two competent developers may implement different behavior from the same requirement. Arav’s diagnostic question is: “Given only the requirement, could two competent developers disagree about exactly 2.00 seconds?” If yes, specify the intended result in the requirement itself.
In an author-reported comparison across ten seeds, three runs converged under the ambiguous wording. After the boundary decision was moved into the requirement, all ten reportedly converged on the first sweep; seven still required the zero-gap repair. The point is not to adjust tests until runs pass, but to make behavior-defining decisions explicit so both implementation and verification have a clear target.
Rank #4
What Arav’s reported results do—and do not—show
Arav reports 967 runs across three sweeps: roughly eight in ten passed integration and system tests, and roughly six in ten passed all stages, including unit tests. He separately reports 390 runs in a fourth sweep, with approximately the same rates after process hardening and making two tasks harder. He says the runs used a small, inexpensive model and frames the results as a performance floor.
Free tools Windows power users keep installed
One-click scans. No signup required.
These are the author’s reported outcomes for the example tasks, not an independently reproduced evaluation. They do not demonstrate that withholding criteria catches more real defects than tests written with access to the code, or that automatically refining criteria makes tests sharper. Arav leaves both questions open and says they require formal proof. The reported convergence rates should not be treated as answers to them.
Best Value
For the radar task, Arav reports that the run took under a minute and fewer than ten model calls. That is a description of his run, not a general time or call-count expectation for this approach.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Verification cannot replace validation
Verification asks whether code meets the written standard. Validation asks whether that standard captures the behavior that should actually happen. Hiding acceptance criteria from the coding agent can strengthen the independence of verification; it cannot decide whether the criteria are right.
A domain expert therefore needs to approve the specification and remain involved as it changes. That matters especially in advanced driver-assistance systems, where it is difficult to enumerate edge cases and operational design domains. Arav points to design-of-experiments principles rather than brute-force coverage as a way to think about test selection, but no method removes the need for human judgment about intended behavior.
Recommended Free Tools
Practical checks for applying the approach
- Keep the information boundary real. Give the coding agent requirements, not the acceptance criteria; have a separate testing agent work from the criteria without reading the implementation.
- Write boundary behavior into the requirement. State outcomes for exact thresholds and other cases where reasonable implementers might disagree.
- Interpret failures and passes in context. A failure against independently derived criteria is a finding to investigate; a pass is limited to the criteria, data, and code history actually involved.
- Check that fixtures can trigger each rule. A test suite cannot provide meaningful evidence for a condition its sample data never reaches.
- Keep a domain expert accountable for the specification. Independent testing checks conformity to the standard, not whether the standard reflects the intended system behavior.
Read Gal Arav’s full article, “Towards Spec-Driven Test Automation: Part 2” (published September 30, 2026).
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
