A validation check can pass a broken result when the defect preserves the property the check measures. In a field-extraction example, a count matched even though a regular expression renamed one field. The fix is to test the right invariant—and to use evidence the instrument under test did not produce.
How a count check missed a renamed field
Mahiro Hirakawa describes writing a reconciliation to catch a filter that silently dropped fields. The check compared how many fields the type declared with how many the filter matched. That sounds useful, but it only verifies a count.
As an Amazon Associate I earn from qualifying purchases.
The example includes the field risk-level. The regular expression w+=S+ does not treat a hyphen as part of the field name. It can therefore match level and attach the value to that wrong name. The field is not absent from the output; its identity has changed. Because the filter still emits five fields, the count comparison passes.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Hirakawa’s demonstration reports five fields for both the good and renaming cases, and ten for the duplicate-emission case. These are illustrative outputs, not general statistics. Read Hirakawa’s account on DEV Community.
What different checks can and cannot detect
| Check | What it verifies | Example failure it can catch | What can still pass |
|---|---|---|---|
| Count comparison | Number of declared and matched fields | A duplicate-emission case that changes the count | A rename that leaves the count unchanged |
| Identity comparison | Whether expected field names appear in the output | A missing or renamed field | A wrong value attached to a correctly named field |
| Value validation | Whether observed values satisfy the expected meaning or constraints | A correctly named field with an incorrect value, if the validation covers that error | Errors outside the validation’s rules |
The two example defects preserve different properties. Renaming preserves the number of fields but breaks the set of names. Duplicate emission can leave the set of names present while changing the count. A count-only check and a name-only check therefore have different blind spots.
Compare the output with declared identities
Hirakawa’s proposed improvement is to compare declared field names with the names found in the extracted result and raise an error when a declared name is missing. That detects the illustrated rename because risk-level is absent from the output under its declared identity. A duplicate that changes the count is detectable by the count reconciliation; checking identities also makes the renamed-field failure visible.
Neither check proves that the values are correct. If the filter returns the wrong value under the correct field name, both the count and identity checks can pass. That requires an additional assertion tied to the value’s expected content or behavior.
Use evidence independent of the instrument
A check becomes circular when it uses the instrument’s own summary as the reference for whether that instrument worked. Hirakawa recounts three examples: using a filter’s match count to validate field extraction, a search result to validate a code search, and an API’s comments field to validate a comment counter. In each case, the repair was to bring in a reference the instrument did not generate: a declared record type, a term known to be present, or a second endpoint that enumerates comments rather than merely counting them.
A positive control is useful only if it is independent of the behavior being checked. If the control comes from the same source as the result, it can repeat the same error rather than expose it. Hirakawa puts it simply: “The requirement is independence.”
Independence addresses circularity; it does not establish that every observed value is semantically correct. A separate source can tell you whether the instrument saw what it should have seen, while value-level checks are still needed to test whether what it saw was right.
Rank #4
Choose an invariant by naming the failure
Before trusting a passing check, ask Hirakawa’s question: “which invariant does this reconcile on, and what failure preserves it”. Then build a concrete failure case and ask, “does this catch the bug”.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall- If the failure could rename or omit an item without changing the total, compare identities, not just counts.
- If the failure could duplicate items, check cardinality or uniqueness as appropriate.
- If the failure could retain names but corrupt their values, validate the values against independent expectations.
- Where possible, make the reference evidence independent of the system whose output is being validated.
As Hirakawa writes, “A check only catches failures that break the invariant it reconciles on.” A passing check is evidence about that invariant—not proof that the entire result is correct.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
