A YAML inline comment silently removed part of a success criterion before it reached a verification gate in the open-source tool spec-lane. The gate did what it was built to do: it compared the values it received, and both sides had been shortened to the same text. The check passed. Maintainer Yusuke Shiki’s write-up describes the failure, a control case that exposes it, and a source-aware guard shipped in v0.11.0.
The one line that lost its last clause
The criterion was written as an unquoted YAML plain scalar with an inline comment:
As an Amazon Associate I earn from qualifying purchases.
- ledger has exactly one PhaseGate row # include the negative case tooTo a person reading the file, the comment reads as part of the requirement. It says the criterion covers a negative case. To the YAML parser, everything from the # onward is a comment, so it is not part of the value. In the reported example, using [email protected], the parsed JavaScript value was only:
ledger has exactly one PhaseGate rowThe comment text stayed in intent.yaml on disk. It never appeared in the data the tool worked with. Nothing in that chain looked wrong, because YAML’s comment rule is working as specified. The problem is that the human’s intent lived in a part of the file the parser is designed to discard.
#1 Best Overall
Why the gate passed
spec-lane keeps success criteria in intent.yaml and the matching verification-matrix records in verification.yaml. The gate compares a criterion against a matrix entry. It never reads the raw source text of intent.yaml, so it had no way to see the comment.
The reproduction used a matrix entry that was also shortened to ledger has exactly one PhaseGate row. Both values matched. On the revision immediately before the fix in PR #48, validation and advancement into the verification phase succeeded with exit code 0.
That result shows the gate is consistent. Given matching values, it accepts them. It does not show that the criterion a person wrote was enforced. Shiki is explicit on this point: the fixture’s evidence and negation-test entries were declarations, and the ledger test named in them was never created and run as proof of the intended behavior. A matrix row naming a test is a claim that the test exists and covers the criterion. It is not a record that the test ran.
Recommended Free Tools
The control case: quoting the criterion
Shiki then wrapped the same criterion in double quotes so the comment text became part of the string:
- "ledger has exactly one PhaseGate row # include the negative case too"He kept the shortened matrix entry and the pre-fix CLI. Validation and advancement failed with exit code 3, and the phase stayed at 3_implement. The full criterion no longer matched the shortened matrix entry, so the gate refused to advance.
The comparison below sets the two source forms side by side:
| Source form in intent.yaml | Value the gate receives | Matches shortened matrix entry (pre-fix) | Pre-fix result | Result under v0.11.0 |
|---|---|---|---|---|
Unquoted plain scalar with trailing # ... |
Text before the # only |
Yes | Exit code 0; advanced to verification | Rejected by the source-aware check |
| Same text wrapped in double quotes | Full text, including the comment-like clause | No | Exit code 3; stayed at 3_implement |
Not stated in the author’s account |
The pre-fix behavior is the core of the story. The gate failed closed on the quoted version, but only because the quoted value was longer than the matrix entry. The unquoted version passed for the same underlying reason: the shortened value was the one that reached the gate on both sides.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteThe v0.11.0 fix: check the source before the value arrives
spec-lane v0.11.0 moved the check to the point where intent.yaml is read. As Shiki describes it, the check:
- walks the YAML abstract syntax tree;
- inspects the original source ranges of each node;
- flags unquoted plain scalars that are followed by whitespace and then
#; - rejects candidate values that appear among the parsed success criteria.
Shiki notes that the check does not rely only on an AST comment property. YAML anchor forms can attach a comment to a different node than the one a reader would expect, so a property-only check could miss the case or flag the wrong node. Inspecting the source ranges directly avoids that dependency.
Rank #4
The fix is tracked as issue #45 and PR #48 in the spec-lane repository. The implementation details above come from the author’s description and should be read that way.
The false-positive trade-off
The guard can be wrong in the other direction. If a different criterion has an unquoted plain scalar with a trailing comment, and its parsed text happens to equal a quoted success criterion, the check rejects the document. In that case the quoted criterion was never truncated, but the document still fails.
Shiki describes this as a fail-closed heuristic aimed at a known silent-truncation path. It is not a proof that the author’s full intent has been preserved, and it does not evaluate meaning. The trade-off is deliberate: a noisy rejection is easier to notice than a gate that advances on a shortened criterion.
Best Value
What a passing gate does and does not establish
Three separate things can each look like confirmation, and none of them is enough alone:
- Schema validity. A schema validator checks the structure of the value it receives. As Shiki puts it, a schema validator can validate the structure it receives. It cannot recover text that the parser discarded before validation began.
- A matching matrix row. A row that names a test shows a claimed link between a criterion and evidence. It does not show the test was written or executed.
- A zero exit code. A successful command means the gate’s checks passed on the values it was given. It does not mean the property the human cared about was demonstrated.
The lesson Shiki draws is about the order of work. When implementation is handed to a coding agent, the definition of done should exist before the implementation does, and that definition should survive the path from source file to gate intact. Shiki states this as: “When I hand implementation work to a coding agent, I want the definition of ‘done’ to exist before the implementation does.”
Practical steps for criteria with # in them
- Quote any success criterion that contains
#, so the parser keeps the full text. - Keep explanatory notes in a separate comment on their own line rather than trailing a criterion.
- Check the parsed value of each criterion before relying on it. A quick script that prints the parsed strings will show anything that was cut.
- Treat a passing gate as evidence that the recorded values agree, and check separately that the tests named in the matrix exist and have actually been run.
- If you use spec-lane, upgrade to v0.11.0 or later, and expect occasional rejections of documents that contain an unquoted commented scalar matching a quoted criterion. Quote the criterion to clear them.
The underlying point is narrow but general: a verification gate can only be as faithful as the values that reach it. When a human’s intent passes through a parser, the safe place to check is the source.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsSource: Yusuke Shiki’s write-up of the spec-lane reproduction and fix. The YAML behavior described here follows the YAML 1.2.2 specification, revision dated 2021-10-01, which defines # as the comment indicator and describes plain and quoted scalar styles.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
