A payment amount can be correct while the verification answer is wrong. In a small synthetic benchmark published September 27, 2026, both hosted models got every amount right, but one still misclassified three completed checks as incomplete. That distinction matters: a completed search with no matching receipt supports “zero observed,” while a timed-out search says the check did not finish.
What the benchmark asked models to verify
The task supplied explicit rules and a fictional payment log. For each case, a model had to return three things: verified received cents, whether verification was complete, and the IDs of supporting records. The task was not to process a payment or inspect a real account; it tested whether a model could interpret supplied records under a defined contract.
As an Amazon Associate I earn from qualifying purchases.
The rules required a matching invoice, recipient, and currency; use of the latest timestamp; deduplication by transaction ID; and subtraction of completed refunds only. Customer statements and quoted provider-looking JSON were untrusted inputs, not proof of payment. The cases varied evidence such as pending versus completed status, mismatched targets, duplicate versus distinct transaction IDs, partial versus full refunds, stale versus current records, and complete versus partial coverage. The author describes 12 counterfactual pairs scored as 24 cases, with four families reusing the same successful-payment control, for 21 unique payloads. Read the September 27, 2026 article by Soccer skills Freestyle.
Three separate answers, not one
- Amount: How many cents are supported by qualifying records after applying the payment and refund rules?
- Coverage: Did the verification finish sufficiently to support an answer?
- Evidence: Which records support the result, including the latest relevant check?
An amount-only score can miss a coverage error. A completed check that finds no matching receipt verifies zero observed amount. A timeout or incomplete check does not establish that the account has no payment; it establishes that verification did not complete.
#1 Best Overall
- With Square Terminal, you can ring up sales, accept payments, and print receipts, all with one device. Use it at the counter or ring up customers anywhere in your store.
- Accept all major credit and debit cards and pay one low rate with no hidden fees and no long-term contracts.
- Process chip cards in just two seconds.
- Get your money as soon as the next business day.
- Use it cordlessly with the built-in battery, designed to last all day.
What the hosted pilot found
The article reports hosted Kaggle pilot results for two models on the same task version. These are counts from this particular synthetic test, not estimates of general reliability.
| Model | Exact answers | Amounts | Coverage | Evidence | Fully correct pairs |
|---|---|---|---|---|---|
| Gemini 3.7 Flash | 24/24 | 24/24 | 24/24 | 24/24 | 12/12 |
| Claude Haiku 4.5 | 21/24 | 24/24 | 21/24 | 24/24 | 9/12 |
According to the article author, Haiku’s three errors were the wrong-invoice, wrong-recipient, and wrong-currency cases. In each, the records represented a successful, complete check, but the payment did not match the requested target. Haiku correctly excluded the amount, yet marked verification incomplete. Under the task’s rules, the check was complete and had found no matching receipt. The mistake was about coverage status, not a fabricated payment claim.
Rank #2
- With Square Handheld, you can accept payments, take tableside orders, or scan barcodes anywhere. With a slim design and comfortable grip, the POS is easy to carry in your palm or pocket. Square Handheld is designed to withstand water splashes and dust. Add an optional protective case for accidental drops. A long-lasting battery and offline payments let you keep selling.
- Slim, pocketable, and lightweight so you can accept payments wherever your customers are.
- Take tableside orders, bust lines, or use the built-in barcode scanner, all with one sleek device.
- A battery that can power through your shift and offline payments let you keep selling, even if your internet is down.
- Accept all major credit and debit cards and pay one simple rate with no hidden fees and no long-term contracts required.
Local runs show why exact-answer scoring matters
The author also reports a separate local pilot. Its execution setup differed from the hosted runs, so these figures should not be used to rank overall model quality.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minute| Model and local configuration | Exact answers | Amounts | Coverage | Evidence | Overclaims | Fully correct pairs |
|---|---|---|---|---|---|---|
| Llama 3 8B, Q4_0 | 13/24 | 16/24 | 21/24 | 22/24 | 8 | 3/12 |
| Qwen 3.5 9B, Q4_K_M | 21/24 | 22/24 | 22/24 | 23/24 | 2 | 9/12 |
An always-zero baseline got 10/24 amount answers correct (41.7%), as reported by the author. That baseline illustrates why amount accuracy alone needs context: a simple answer can sometimes match a case without correctly interpreting coverage or evidence.
Rank #3
- Accept all major credit and debit cards and pay one low rate
- No hidden fees and no long-term contracts
- Mobile card reader that accepts payments anywhere & anytime
- Use the free SumUp App on your smartphone or tablet to start accepting transactions
- Simply pay 2.6% +10 per in-person transaction
How the pilot was run—and what it cannot establish
The article says Gemini 3.7 Flash and Claude Haiku 4.5 completed the identical task v1 on Kaggle on September 27, 2026. Hosted runs used Kaggle’s kaggle-benchmarks 0.6.1 proxy and a fresh chat for each case; the notebook supplied no custom generation settings. A selected hosted Qwen run failed twice with HTTP 429 before a model turn was recorded, so it received no score.
For the local pilot, the author reports using Ollama llama3:8b (Q4_0) and qwen3.5:9b (Q4_K_M), with fresh conversations, temperature 0, seed 42, a 4,096-token context, a 256-token output cap, and JSON-schema formatting; thinking was disabled when supported. Each case had one generation. The author says the fixtures, instructions, and settings were hashed before local execution, gold answers were entered and cross-checked with a rule interpreter, six offline checks covered scoring and boundary cases, and raw responses and grades were retained. These are the author’s reproducibility claims; Kaggle task and run artifacts are linked from the article, but their contents have not been independently verified here.
Rank #4
- An intuitive interface to easily accept payments and manage your sales.
- Strong, reliable Wi-Fi connection. Free SIM card and mobile data so you can process payments anywhere.
- Great battery capability with an additional charging station.
- A truly portable device. Stay in control of your business, wherever you go.
- Support when you need it. Get in touch with our US-based support through phone, email and chat.
- The 24 scored cases are deliberately correlated, and four case families reuse a successful-payment control.
- A perfect result means only that this pilot found no failure for that model on these cases; it does not establish production reliability.
- The local models used different quantizations and execution conditions from the hosted models, and one generation per case cannot show run-to-run variation.
- All people, records, and organizations in the task are fictional. It does not authenticate real tools, inspect accounts, move money, or reproduce any payment provider’s complete settlement rules.
What a useful payment-verification evaluation should report
This benchmark’s clearest lesson is about the scorecard. A model response should be judged on at least three independent axes, alongside an all-fields exact score:
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errors- Amount correctness: Did it apply payment state, target matching, deduplication, and completed-refund rules correctly?
- Coverage correctness: Did it distinguish a complete empty result from incomplete verification?
- Evidence correctness: Did it cite the relevant record, especially the latest applicable check?
- Exact-answer correctness: Were amount, coverage, and evidence all right together?
Keeping these dimensions separate makes a subtle error visible: “the amount is zero” and “the check did not complete” are not interchangeable statements. In systems that support consequential decisions, an evaluation must test the status of the evidence-gathering process as carefully as the numeric result.
Quick Recap
Best Value
- Effortless payments and printing: Accept card payments and print payment receipts on the spot with the built-in 40 mm thermal printer.
- Faster sales processing: Use pre-set menus and catalogs to make transactions faster and smoother for you and your customers.
- Reliable and portable: Featuring a 6.5" HD touchscreen made from Corning Gorilla Glass and a powerful battery that lasts all day.
- Seamless connectivity: Stay connected with free mobile data and WiFi, ensuring uninterrupted transactions.
- Real-time payment tracking: Monitor payments and issue refunds right from your device, so you're always in control.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
