Counting failed connections as incorrect answers can make an AI evaluation look decisive while obscuring what it actually measured. Jordan Liu’s September 21, 2026 article on DEV Community makes the case for separating whether a response was gradeable from whether it was correct. Its example is a hand-planted fixture—not a live test of models or endpoints—so its figures illustrate the scoring method, not anyone’s real-world performance.
Why a single pass/fail score can mislead
An evaluation may produce no usable answer for several different reasons: a connection can fail, a response can be empty, generation can stop partway through, or the output can violate a required schema. Those outcomes are not equivalent to a valid answer that simply gets the question wrong.
As an Amazon Associate I earn from qualifying purchases.
If all of them are folded into one failure count, a score cannot show whether a change came from answer quality or from the path that delivered the answer. Liu puts the distinction plainly: “HTTP 200 is a door. It is not a grade.” An HTTP success status does not establish that a response contains a gradeable answer.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Classify the response before grading its meaning
Liu’s example assigns each envelope one of six labels, moving from delivery and format checks toward semantic scoring:
#1 Best Overall
- Wiley
- Language: english
- Book - storytelling with data: a data visualization guide for business professionals
- Drop: a refused or reset connection, a client timeout, HTTP 429, or HTTP 503.
- Empty: HTTP 200 with no choices, or content that is null or empty.
- Truncated: a
finish_reasonoflength, or JSON cut off mid-value or mid-key so it cannot be parsed. - Schema: parseable JSON that omits a required field.
- Wrong: valid JSON whose answer does not match the expected value.
- Right: a gradeable response with the expected answer.
This ordering matters: first establish that there is a response that can be evaluated, then score its answer. A parseable object missing a required field is a schema failure in this scheme; it is not a wrong semantic answer unless it reaches the grading stage.
What the chart’s percentages actually mean
Liu’s Python example uses 24 in-memory envelopes, with four deliberately planted examples for each label. The arithmetic is useful for understanding the denominators, but it does not measure an API, model, or production workload.
| Measure | Fixture result | What it counts |
|---|---|---|
| Naive pass rate | 16.7% (4 of 24) | Correct answers divided by all planted envelopes. |
| Yield | 33.3% (8 of 24) | Gradeable wrong and right answers divided by all planted envelopes. |
| Accuracy-on-yield | 50% (4 of 8) | Right answers divided by gradeable right plus wrong answers. |
The fixture contains 16 ungradeable envelopes: four each labeled drop, empty, truncated, and schema. So the 50% accuracy-on-yield result means four of the eight gradeable responses were right. It does not mean a real service answered correctly half the time. As Liu writes, “The percentages are the fixture talking, not a vendor scoreboard.”
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallWhat to report in an evaluation
A report that keeps the stages separate is easier to interpret and diagnose. Liu’s proposed measures and comparison axes are:
- Yield: the share of calls that reached a semantically gradeable answer.
- Accuracy-on-yield: the share of gradeable answers that were right.
- Failure mix: the counts or proportions of drops, empties, truncations, schema failures, wrong answers, and right answers.
- Run conditions: whether the figures came from planted or live data, which retry and repair rules were used, what was logged, and whether candidate and judge endpoints were shared.
Always state the denominator. Accuracy-on-yield excludes ungradeable responses; yield includes all calls in its denominator. Reporting only one of those numbers can hide a materially different picture.
Retries and repairs are policy choices
Liu recommends retrying drops and empties, capping attempts to repair truncated and schema responses, and scoring wrong answers without retrying them. That is the author’s suggested policy, not a result from a controlled comparison of retry strategies. A different policy can change both the observed failure mix and the eventual scores, so document it rather than treating the final chart as policy-free.
Keep transport failures distinct from answer failures in logs. Liu suggests recording HTTP status, latency, finish reason, bytes, label, and then the answer. The article’s sample curl probe checks the number of choices, finish reason, and serialized response size. These details help explain what happened when a call does not produce a gradeable answer; a status code alone cannot do that.
Free tools Windows power users keep installed
One-click scans. No signup required.
What the example does—and does not—establish
The article’s evidence is its hand-authored fixture and its arithmetic. It reports no live endpoint runs, model comparisons, or measured production reliability. Liu also says the percentages are not a vendor scoreboard and declines to name models, quote quotas, or claim uptime. The fixture therefore demonstrates how categories and denominators can be separated; it cannot show how often any specific model or service drops, truncates, or answers correctly.
Best Value
Liu cautions that a free or shared model path is not a latency SLA, a deterministic-output guarantee, or a substitute for a held-out human grade. The article recommends separating candidate and judge endpoints where possible, because shared load can couple their latency and contribute to client timeouts. Those are operational recommendations, not findings demonstrated by the fixture.
The author identifies the article as prepared as part of MonkeyCode product outreach and uses MonkeyCode’s free model access and free server option as an example. That disclosure is relevant context when weighing the example; it does not establish service quality, availability, or current commercial terms.
Quick Recap
How to apply the distinction to a real evaluation
- Define the labels before running calls. Specify what counts as a transport drop, empty response, truncation, schema failure, wrong answer, and right answer for the response format you expect.
- Retain enough response detail to classify failures. Capture status, latency, finish reason, response size, the assigned label, and answer content as appropriate for your data-handling rules.
- Write down retry and repair rules. Record which labels trigger another call or a repair attempt and any attempt cap; do not silently change these between compared runs.
- Report denominators and the failure mix. Give yield and accuracy-on-yield separately, alongside counts by label and the run conditions.
- Use live evidence for live claims. A planted fixture can check whether your classifier and arithmetic behave as intended, but claims about endpoint reliability or model performance require actual evaluation runs under stated conditions.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problems
