Recommended Free Tools
A model tool-call test flagged a model for sending the wrong arguments. The model was following the provider’s schema. The harness’s expected object was the part that was wrong. That is the core of a September 6, 2026 DEV Community postmortem by an author displayed as “Self-Correcting Systems,” and it applies to any test that compares model output with a frozen fixture.
What happened
The author describes a harness that prepared an exec call before the model ran and froze it. It told the model to send exactly that JSON object, then compared the model’s actual tool arguments with the frozen one. The expected object had a single key, command. The model’s call contained both intent and command, and the harness recorded EXEC_ARGUMENTS_MISMATCH.
As an Amazon Associate I earn from qualifying purchases.
The author checked the sandbox exec tool in a locally installed compiled artifact of @truefoundry/[email protected]. They report that it requires intent and command, with cwd and env optional. These details come from the author’s own inspection. They are not a confirmed statement about the current upstream schema, so check the schema in your own version rather than assuming it matches.
The model’s call satisfied the provider’s required fields. It violated the harness’s instruction to send exactly one key. The post says the model’s behavior was compliant with the schema and the fixture was the defect.
#1 Best Overall
Why a mismatch can’t tell you who is wrong
The post puts it this way: “A mismatch establishes difference, not which operand is authoritative.” A comparator sees two values that differ. It cannot know whether the actual value broke a rule or the expected value did. Three separate things are easy to conflate:
- Provider protocol requirements: what the tool schema accepts and demands.
- Harness or run policy: narrower rules you impose, such as “send exactly this object.”
- Fixture validity: whether your expected object is itself acceptable under the provider schema.
Here the third item failed silently. The harness treated its own policy, built on an invalid fixture, as if it were the provider’s contract, and charged the difference to the model.
The fix, and what stayed the same
As reported, commit 0220a27 added a constant, CANDIDATE_VERIFICATION_INTENT = 'Run candidate verification', to the expected object next to command. A new gate requires exactly those two keys and the fixed intent value. The comparison still parses the actual JSON and compares canonical JSON bytes, so key order does not matter. The comparator was not changed; the expectation and gate were.
The author contrasts this with the tempting alternative of loosening the check, for example by ignoring extra keys. Strictness can be legitimate. A run can demand something narrower than the provider allows. The condition is that the stricter rule is labeled as your policy and that your fixture is valid under the provider schema first.
Checks to run on your own harness
- Where did the expected object come from: a provider schema, harness policy, or an assumption about the API?
- Does the expected object validate against the active provider schema before the model runs?
- Does the failure report separate schema noncompliance from violation of run-specific policy?
- When you correct a false failure, does the repair keep the comparison control intact rather than weakening it?
- Does the receipt record the contract and version that governed the verdict? This is an inference from the post’s drift concern, not a feature of the described harness.
The post’s own advice: if a comparison sits between you and a model, read the schema you are comparing against and check that your expected object satisfies it.
Design approaches compared
The post compares no products. These are conceptual axes for judging a harness.
Rank #4
| Axis | Weak design | Stronger design |
|---|---|---|
| Schema authority | Required fields duplicated in harness constants | Required fields read from the active provider schema |
| Policy separation | Provider rules and run rules mixed | Narrower per-run constraints applied as a distinct layer |
| Fixture preflight | Expected object assumed valid | Expected object validated before the model is invoked |
| Failure attribution | One generic mismatch code | Result says provider-schema violation or harness-policy violation |
| Drift handling | Stale expectations show up as model errors | Schema changes detected and reported as such |
The remaining weakness: a hardcoded key count
The author acknowledges that the new gate’s argumentKeys.length !== 2 check bakes in two things at once: today’s provider-required fields and the harness’s decision to reject optional ones. If the provider adds a required argument, the harness could reject a compliant call until someone updates it. The proposed direction is to derive required fields from the actual schema and layer narrower harness rules on top. At the time of writing, the author says this was not built.
This run did not verify anything
Fixing the expectation did not make the run a success. The same receipt also recorded EXEC_RESPONSE_SHAPE_UNEXPECTED, and the sandbox had no JavaScript runtime. The author says candidate verification was therefore not established, and points to a separate post for the runtime issue. Treat it as a failed verification run with one false failure removed, not as evidence that the candidate passed. These receipt details are the author’s account; they were not independently inspected.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
