Yes. An LLM evaluator can return a well-formed score or pass/fail verdict even when the evidence behind it does not support the claim. A 2026 exploratory preprint documents 13 reproduced measurement defects across seven author-selected open-source evaluation tools. Its cases show how results can be undermined by missing evidence, the wrong unit, a proxy, stale state, reversed threshold arithmetic, or a check aimed at the wrong target. They are documented mechanisms, not an estimate of how often evaluation tools fail.
What the study examined
DriftproofHQ’s preprint, Reported, Not Measured: An Empirical Study of Measurement Defects in LLM and Agent Evaluation Tools, is version 1, dated 30 September 2026, and identified as not peer reviewed. The author says they inspected scoring and gating code from 14 to 26 September 2026 in seven open-source tools or evaluation setups: the agent-skills repository’s eval harness and a reference gate script, NVIDIA SkillEvaluator, DSPy, LangSmith, DeepEval, Harbor, and MLflow. The tools were selected by the author’s judgment, not sampled to represent the wider field. Read the preprint and its accompanying materials.
As an Amazon Associate I earn from qualifying purchases.
The author reports reproducing 13 defects, 12 offline, and filing each upstream with a public reproduction. Twelve cases had a proposed fix. At the preprint’s 30 September cutoff, five fixes had merged, one was approved, and the rest were open. A follow-up write-up dated 6 October reports that the MLflow fix had since merged; that later status is the author’s report, rather than an independently verified upstream check. Preprint; Follow-up write-up.
Free tools Windows power users keep installed
One-click scans. No signup required.
Six ways a reported result can fail to measure its claim
The paper organizes its 13 cases into six mechanisms. They describe how a result can lose its connection to the evidence or behavior it is supposed to represent—not how common each mechanism is.
#1 Best Overall
1. Missing evidence is treated as a measurement
A tool produces a score or verdict even though it lacks evidence required to support that result. A syntactically valid number does not fill the evidentiary gap.
2. The result is attached to the wrong unit
A reported outcome is bound to the wrong item or unit of analysis. Even if the calculation itself is sound, attributing it to the wrong run, task, or other unit makes the result misleading.
3. A proxy is credited as the target behavior
The evaluator measures a stand-in and reports it as though it measured the behavior the evaluation is meant to assess. A proxy can be informative, but the report should not silently turn it into the target itself.
4. Stale state is presented as current
A decision uses outdated state while appearing to describe the current state. A result’s timestamp or freshness therefore matters alongside its value.
5. Threshold arithmetic reverses the intended comparison
A sign or direction error changes which results pass a gate. The paper’s MLflow example shows how relative-improvement logic can go wrong when a baseline is negative.
6. The check targets something the artifact did not claim
A validation checks a different target from the one the evaluated artifact claims to satisfy. Passing that check does not establish the original claim.
Why a negative baseline can invert an evaluation gate
The follow-up write-up describes a bug in MLflow’s MetricThreshold, a gate intended to assess relative improvement over a baseline. The calculation divided the change by the signed baseline. When that baseline was negative—as R² can be—the sign could reverse, so a worse candidate might pass while a better one failed.
Recommended Free Tools
The author’s 6 October write-up says the merged fix uses the magnitude of the baseline and adds 26 regression cases covering positive and negative baselines and both metric directions. This is a concrete example of why reviewers should inspect the direction and denominator in relative-threshold logic, not just whether the gate returns a clean pass/fail result. Read the author’s follow-up write-up.
Why more runs cannot repair an invalid measurement
Statistical methods can characterize uncertainty in a valid measurement. They cannot make unsupported evidence valid. If a score comes from missing evidence, is bound to the wrong unit, substitutes a proxy for the target, uses stale state, or applies a check to the wrong claim, repeating the evaluation or narrowing a confidence interval does not repair that underlying mismatch.
The preprint includes published measurements as illustrations of evaluation challenges, not as tests of its own proposal: a pass/fail check and a text-injection evaluation that interpreted the same task differently under different treatments; one evaluation that returned pass eight times and fail twice at a single commit; and two release-day comparisons in which most cells did not separate. These examples should not be read as general failure rates for evaluators.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What to inspect before trusting an evaluator’s score
The six mechanisms suggest practical review questions. These are implications drawn from the cases, not interventions the study independently tested.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11- Trace the evidence: Identify what evidence the score or verdict actually consumed. Confirm that it supports the stated claim.
- Check the binding: Verify that evidence is attached to the correct run, item, or other unit of analysis.
- Name the target: Determine whether the evaluator measures the claimed behavior or a proxy, and make that distinction explicit.
- Check freshness: Confirm that the state used for the decision is current for the evaluation being reported.
- Review the gate math: Inspect comparison direction, denominator, and behavior with negative baselines where those are possible.
- Match the check to the claim: Ensure the validation target is the one the evaluated artifact actually claims.
- Keep validity separate from precision: More repetitions can help characterize variation, but cannot fix a measurement that lacks a valid connection to its target.
When comparing evaluation tools or record designs, useful dimensions include whether each result records its evidence, unit, target, and state freshness; how gates implement direction and thresholds; and whether reproductions or regression cases are publicly available. Those dimensions support a more meaningful comparison than treating a score as self-explanatory.
Best Value
What the proposed evaluation record does—and does not—establish
The paper proposes nine evaluation-record fields, each paired with a check, and maps its defect cases to fields and checks that might have exposed them. The authors describe this mapping as a design argument that has not been tested. It is a proposal for making evidence and evaluation claims easier to inspect, not a validated framework or proof that adopting the fields prevents defects. See the preprint.
How far to generalize the 13 cases
The study is exploratory: it examines code selected by one author’s judgment, and its authors explicitly describe the findings as a lower bound on mechanisms found where they looked, not a prevalence estimate. The counts—13 reproduced defects across seven selected tools or setups, with 12 reproduced offline—describe this set of cases. They do not show what proportion of evaluation tools, individual evaluations, or published results are defective.
Likewise, the reported fix status is time-bounded. The preprint gives counts as of 30 September 2026; the MLflow merge is a later status reported in the 6 October author write-up. Neither count should be treated as a live inventory of upstream fixes.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
