Recommended Free Tools
A green test run tells you that the checks in the suite passed under the inputs, environment, and assumptions they encode. It does not tell you that the software will hold up against failures nobody wrote a check for. Derek Wang’s essay “A test system that can say ‘I don’t know’ is worth more than one that says ‘passed'” (DEV Community, 2026) argues that a suite should be built to notice when a failure matches nothing it already knows, and to record that gap rather than bury it inside a pass count.
What a green result actually certifies
A passing check certifies one thing: a specific input, in a specific environment, produced the output the author expected. Stacking more checks on top raises the count, but the count measures how many situations were encoded, not how much of the failure space was covered. Two suites with identical totals can differ sharply in what they would notice going wrong.
As an Amazon Associate I earn from qualifying purchases.
The useful question is therefore not “did everything pass?” but “what would this suite have reported if something unfamiliar had happened?” A suite that can only answer “pass” or “fail” against its own checklist has no way to say the third thing that matters most: this result matches nothing we understand.
Known failures, unmatched failures, and unresolved outcomes
Wang’s central distinction separates three kinds of result that a plain pass/fail report flattens into one:
- Known failure. The run matches an entry the team already has: a recognized shape, a diagnosed root cause, and a repair recipe.
- Unmatched failure. The run fails in a way that matches no entry. The essay treats this as an unknown and records it for investigation.
- Unresolved outcome. A result the team has not yet classified or diagnosed, which needs a decision rather than an automatic label.
The unmatched category is the informative one. An unmatched failure is not just another red mark; it indicates that the team’s map of failure modes is incomplete. A suite that hides it under a generic “failed” label, or worse under an overall pass, discards the most useful signal it produced.
How the fulltest workflow is described
Wang describes a workflow he calls fulltest with three elements. They are the author’s described practices, and each depends on people maintaining them.
1. Classify before judging
Each run is compared against a ledger of failure portraits. A portrait records the known shape of a failure, its root cause, and a repair recipe. If a run matches a portrait, it is reported as a known failure with a known response. If nothing matches, the run is treated as an unknown rather than forced into the nearest category.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
2. Grow the ledger from incidents
Entries are added from three sources, according to the essay: unmatched failures once they are investigated, issues that have been diagnosed and whose root cause is understood, and repair commits. The author frames this upkeep as organizational learning that happens between runs. A single execution does not complete the ledger on its own.
3. Raise the baseline after improvement
Each run is compared with a recorded baseline. When results get stronger, the stronger state is written down so the new floor cannot be quietly lost. When a regression occurs, the useful output is not just “the numbers dropped” but which failure class has returned. A baseline is only meaningful together with the categories and scope it counts; a single percentage with no stated scope tells a reader very little.
Comparing reporting models
The table below sets a conventional pass/fail report beside the ledger-based model Wang describes. The right-hand column reflects the essay’s stated design; it is not a measurement of any tool on the market.
| Question a reader needs answered | Conventional pass/fail report | Ledger-based model described in the essay |
|---|---|---|
| Does a failure match something already understood? | Not reported; a failure is simply a failure | Yes, each run is matched to a failure portrait |
| Is an unexplained result visible as its own category? | Usually folded into “failed” | Recorded as an unmatched failure and investigated |
| Is an unclassified outcome distinguished from a known one? | Not stated | Yes, as an unresolved outcome that needs a decision |
| Is the failure history tied to diagnosis and repair? | Not stated | Yes, each portrait links cause to repair recipe |
| Are baseline changes explicit? | Depends on the team | Improvements are recorded so the new floor persists |
| When results regress, what is identified? | The failing test names | The failure class that has returned |
| Is there evidence the suite caught past incidents? | Test count is the usual evidence | Prior incidents are tracked as portraits and checked on later runs |
The figures Wang reports, and what they do and do not show
The essay includes project figures from the author’s own system. They are reported as original data by Derek Wang in 2026. They have not been independently audited, and they are not a representative benchmark for other teams or systems.
- 10 seconds. Wang reports that the full regression run took about ten seconds on his project.
- 50 scenarios across six suites. Wang reports coverage of contracts, idempotency, retrieval, regression, resilience, and end-to-end tests.
- 22 hidden HTTP-500 errors. Wang says the full suite caught all 22 before release. The essay does not specify how these were counted, so read the figure as a single-project account.
- 62/62 integration checks. Wang says an integration test was fully green during a later production incident, and that all six suites passed at that time.
The last figure is the one that matters most for this article. A suite can be fully green and still be wrong about the system it is testing, as the next section shows.
The incident that passed every check
The essay’s most useful example is a production failure that escaped an all-green suite. A third-party endpoint returned a malformed response shape during a narrow time window. The error then propagated through the chain of services that depended on it.
Rank #4
The team reportedly could not reproduce the anomaly reliably, because it depended on a particular distribution of data that the test environment did not generate. The boundary condition had never been represented, so no check could fail on it. Wang argues this was an unmodeled boundary rather than a simple shortage of tests. Adding more checks of the same kind would not have covered it.
His proposed response has two parts: keep regression checks, and add failure-mode analysis together with resilience in parsing and degradation. In practice that means asking how the system should behave when the shape of an upstream response is unexpected, not only whether it behaves correctly when the shape is expected.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11When a failure cannot be reproduced
Reproducible testing answers whether a known behavior still works. Failure-mode analysis asks a different question: if a failure occurs that cannot be reliably reproduced, can the system tolerate it, contain it, or degrade safely? Teams applying this idea usually work through three questions for each dependency:
Best Value
- Tolerate: does the parser reject or handle a malformed shape without crashing or corrupting state?
- Contain: does the error stop at the boundary, or does it multiply across dependent services?
- Degrade: when the dependency misbehaves, does the system fall back to a defined, safe behavior that users can understand?
None of these questions requires reproducing the original anomaly. They require deciding in advance what safe behavior looks like, which is exactly the kind of knowledge a failure ledger is meant to accumulate after each incident.
What a ledger and a baseline cannot do
Wang’s framing is careful on this point, and readers should keep the same caution. An accumulating ledger does not make a system comprehensive. It records the failures the team has seen or imagined and how it responded. A baseline alone does not establish software quality; it only shows whether a measured state has moved. An unmatched failure is valuable because it reveals a gap, but closing that gap still requires people to investigate, decide, and update the portraits.
Evaluation design can also reward the wrong behavior, a point made in a separate commentary on evaluation scoring (published on Claims · The Evaluating Self). In a scoring rule that awards a point for a correct answer and nothing for abstaining, guessing can score higher in expectation than an honest “I don’t know.” That commentary concerns the scoring rule itself and does not establish how much the incentive explains any real system’s behavior. It is an analogy for testing, not a finding about test suites: a report that never allows “unknown” will tend to produce confident-looking results whether or not they are justified.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →The practical takeaway for a test suite
Judge a suite by what it can say, not only by what it has passed. Check whether it separates known failures from unmatched ones, whether unresolved outcomes are visible, whether baseline changes are written down with their scope, and whether it can point to the failure class that returned after a regression. Then ask whether any untestable or hard-to-reproduce failure has a tolerance, containment, or degradation plan. A suite that can report “I don’t know” in those terms gives a team information it can act on, which a suite that only reports “passed” cannot.
Derek Wang’s essay is one practitioner’s account from one project, and its figures should be read that way. Its argument stands on its own terms: a green result describes what was encoded, and the failures worth worrying about are the ones nobody encoded yet.
Quick Recap
|body_html_end_marker_removed
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
