Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →A model can classify every cash-related event correctly and still fail the task if its answer is wrapped in Markdown or breaks the required JSON schema. In Hongli Wang’s small synthetic benchmark, output format was not presentation polish: it was part of the answer the evaluator accepted.
What did the benchmark test?
Wang describes 16 fictional cases involving nine delivery states. For each case, a model had to return one JSON array with exactly four keys: id, status, cash_eligible, and revenue_usd. The prompt included the frozen rubric and case narratives, while the separate gold labels were withheld.
As an Amazon Associate I earn from qualifying purchases.
The primary score covered 48 exact field comparisons. IDs were validated and used to match rows, rather than counted as scored fields. Whether rows appeared in the requested order was reported separately and did not change the primary score.
Cash eligibility was defined by a synthetic rubric
Under this benchmark’s rules, qualifying receipts included bank cash received, fiat balances that were settled and withdrawable, and received prepaid cards with documented broad usability in Australia. Points and unconverted ETH did not qualify. If a qualifying non-USD receipt had no documented conversion, its USD amount remained unknown. These are task definitions, not claims about market value or legal treatment.
#1 Best Overall
Unknown, false, and zero mean different things
The task depended on preserving evidence boundaries. An asset with unknown eligibility is not the same as an asset known not to qualify, and neither state is equivalent to an evidenced amount of zero. In the reported output, nine unknown received-asset eligibility values became false. That is a substantive classification error: missing evidence was converted into a negative conclusion.
Why could a correct answer fail?
The evaluator required machine-readable output matching a strict contract. It rejected prose, Markdown code fences, missing or duplicate cases, duplicate keys, extra fields, invalid types, and non-finite amounts. A human might recognize a fenced JSON array as understandable, but a consumer expecting the exact contract cannot necessarily accept it.
Wang reports that an initial interactive run of Gemini 3.7 Flash received a strict score of 0% because the response was fenced in Markdown. On a preserved copy, removing only the fence yielded a 48/48 field-match diagnostic. That diagnostic helps isolate the formatting failure; it was not the recorded task result and does not turn the rejected response into a passing submission.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →What were the reported benchmark results?
The figures below are reported by Wang in his October 5, 2026 account. Formal task results and schema assertions are described as Kaggle platform-reported; the author says they were not independently recomputed. The interactive and offline figures are author-reported.
| Run | Reported outcome | What the result represents |
|---|---|---|
| Gemini 3.7 Flash, formal v1 task build | 100.0% strict | Kaggle platform-reported result for one formal task execution. |
| GPT-5.4 mini, formal v1 task run | 97.9% strict | Kaggle platform-reported result for one formal task execution. |
| Gemini 3.7 Flash, first interactive run | 0% strict; 48/48 field-match diagnostic after removing the Markdown fence from a preserved copy | Author-reported evaluator result and separate diagnostic, not the recorded task score. |
| Quantized Qwen 3.6-35B-A3B, offline CPU run | 35/48 field matches; 6/16 complete cases | Author-reported preparation result using a different recorded system-instruction context; not part of the leaderboard. |
The results should not be collapsed into a single statement about general model reliability. In particular, Wang’s summary is apt: “One perfect platform result does not establish repeatability.”
What do the scores establish—and what do they not?
They show performance on this task and contract
The two formal results show that the submitted outputs received the stated strict scores on this particular rubric-driven task. The distinction between strict scoring and a field-match diagnostic also shows why syntax and schema compliance can matter independently of the underlying labels.
They do not establish repeatability or broad financial reliability
The report describes only three actual Kaggle cloud executions across two distinct models: two formal v1 results and one separate interactive run. The formal and interactive executions used different contexts, and this was not a controlled repeated-seed experiment. The dataset was small, synthetic, and public, with no held-out set or repeated-seed study; public labels may also permit memorization.
Recommended Free Tools
The author additionally reports errors involving successful platform registrations, an untested implementation, and an unverified gift card. Those cases reinforce the narrow lesson: a model should not turn an absent verification step into an established fact. They do not support a broad conclusion about how models perform in real financial workflows.
Evaluator tests check the evaluator, not model capability
Wang reports 23 automated tests for the local evaluator, including single-field mutations and malformed-output checks. Those tests support the described behavior of the evaluator; they are not a measure of general model ability. The report also notes that a narrower completion diagnostic, on its own, misses accepted-to-merged upgrades.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How should a reader interpret this benchmark?
- Read the 100.0% and 97.9% figures as single, platform-reported formal-task observations, not estimates of repeatability.
- Keep the strict score separate from the normalized field-match diagnostic. The latter explains one failure mode but does not replace the recorded result.
- Treat
null, a known negative, and an evidenced zero as distinct values when evidence is incomplete. - Do not generalize a small public synthetic task to real-world finance, legal treatment, or all model outputs.
The primary account is Hongli Wang’s “Cash Evidence Boundaries: Correct JSON Is Part of the Answer”, published on DEV Community on October 5, 2026. The report names the Kaggle Benchmarks SDK, but its reported evaluator checks and interface integration do not broaden what the model results establish.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
