What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Yes: an AI can give the right decision and explanation while returning the wrong policy identifier. In a synthetic support benchmark, a model described the applicable 58-credit policy correctly but put a different policy ID in its structured source field. That mismatch matters whenever software consumes the ID rather than the prose.
What happened in the policy-ID example?
In the case v2-temporal-2-a, the fictional event occurred on June 14, 2026. Policy te-2-a allowed 58 credits and ended June 15 exclusively. Policy te-2-b allowed 73 credits and began June 15 inclusively. The benchmark’s rule made te-2-a applicable on June 14.
As an Amazon Associate I earn from qualifying purchases.
The model’s answer text identified the 58-credit amount and explained that te-2-b did not apply yet. But its structured source_ids field named te-2-b. The answer therefore sounded right to a reader while giving downstream software the wrong evidence identifier. The boundary rule was explicit in the prompt; this was not a case where the dates were absent.
Why can a correct explanation and a wrong ID coexist?
The benchmark separated the natural-language explanation from the machine-readable contract. A response had to provide five JSON fields: decision, source_ids, missing_fields, conflict_ids, and answer_text. The permitted decision values were answer, clarify, and handoff.
Those fields serve different purposes. A person can read an explanation that names the correct rule, while an application may act on the identifier in source_ids. If those parts disagree, a system that trusts the ID could route, audit, or justify the outcome using the wrong policy. Fluency alone does not establish that the evidence reference is correct.
What Support Boundary Bench tested
Author guanguan li described Support Boundary Bench in a Kaggle Benchmarking Challenge post published October 1 and edited October 2, 2026. The policies, products, fees, and customer situations are fictional; the author says no real customer data or actions were involved. The benchmark tests whether a model selects a support decision and supplies evidence that supports it. Read the author’s benchmark post.
The author prepared 10 development cases and froze 30 evaluation cases in 15 pairs. Within each pair, one factor changed, such as evidence order, a required fact, event date, source authority, or an untrusted instruction. Some changes were meant to change the correct output; others tested whether the model kept its answer stable when the changed detail should not matter.
A pair earned a point only when both cases passed every structural check. The reported pair score is therefore passed pairs out of 15, not simply the share of individual responses that looked plausible. Format failures counted against the score; provider failures stopped a suite without producing a numeric capability score. Explanation quality was assessed separately.
What the reported runs show
The post contains historical comparisons and later version 4 results. They should not be blended: the author says the public leaderboard displays version 4, while the earlier rows document the original comparison and a planned replication.
| Run | Valid contract | Structurally correct / assigned | Pairs passed |
|---|---|---|---|
| GPT baseline | 30/30 | 26/30 | 12/15 |
| GPT planned replication | 29/30 | 26/30 | 12/15 |
| Gemini baseline | 30/30 | 30/30 | 15/15 |
| GPT version 4 | 30/30 | 27/30 | 12/15 |
| Gemini version 4 | 30/30 | 30/30 | 15/15 |
The original comparison used openai/gpt-5.4-mini-2026-03-17 and google/gemini-3.7-flash, with the same inputs, prompts, labels, and scoring rules. It used each SDK’s default temperature, no seed, and one attempt per case. The planned GPT replication reused the same 30 cases, so it was a repeatability check, not a new holdout. After date-related failures appeared, the author rebuilt version 4 to correct platform task selection and ran fresh evaluations.
In the original GPT baseline, the decision type was correct in all 30 cases, but only 26 responses passed every structural check; all four failures involved policy dates. In the replication, three temporal responses again explained the right policy while returning wrong source fields, and one response used the invalid enum hand-off instead of handoff. Among valid replication responses, decision accuracy was 29/29, but the case and pair denominators still include the invalid response.
The two GPT runs both passed 12 of 15 pairs, but not the same failures. Two case IDs failed in both rounds while other failures changed. Equal pair totals therefore do not mean identical behavior. Gemini passed all 15 pairs in its original evaluation and in version 4.
What the scores do—and do not—establish
These numbers describe a small, synthetic set with shared templates. The author cautions that the case count, unequal repetitions, and benchmark design do not establish a general ranking between models or predict likely outcomes for real customers. Nor do the results show how often policy-ID errors happen across deployed systems.
Pair scoring captures whether both members of a controlled comparison passed all structural checks, but it does not by itself explain the kind of mistake. A wrong source field and an invalid enum can both cause a pair to fail for different reasons. Reading the component fields and reviewing failures adds information that a single aggregate score cannot supply.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How the author checked the evaluation
An earlier source file hard-coded GPT, so a run labeled Gemini had actually called GPT. The importer detected identical actual model IDs and rejected the comparison. The author kept the extra GPT run and its request cost separate rather than relabeling it. The corrected entry point used platform-injected kbench.llm, and the author says the requested model was checked against recorded evidence.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →For each reported run, the author verified 120 child-file hashes, all 30 recorded prompts, the frozen input, label, and scorer hashes, and agreement between the parent result and independent scoring. The request-cost values in the original table were exported request metrics, not a project invoice.
Best Value
In an October 2 update, the author described reviewing 11 structurally failed responses from baseline, replication, and publication runs individually. The review used AI-prepared Chinese translations and summaries, policy tables, output fields, and suggested judgments; the author then checked each judgment against the conversation and linked decisions to original response hashes. Five reviewed responses had correct explanations and amounts but incorrect policy citations. Five others had date-applicability or explanation errors, including one wrong amount. One identified a policy conflict but used the invalid hand-off enum.
The author characterized this as AI-assisted, non-blind review by one participant—not independent expert validation. Those responses were selected from repeated runs of the same cases, rather than as a representative sample. Full label review and review of the remaining responses were incomplete; the frozen scorer, original outputs, and reported scores were unchanged.
What to validate before relying on a structured answer
The benchmark’s practical warning is narrow but important: do not treat a persuasive answer as proof that its evidence fields are right. The author recommends validating source IDs against the product and event date before downstream software relies on them. In a date-bound policy workflow, that means checking that the cited policy covers the relevant product and that the event falls within its effective interval, including whether the start and end dates are inclusive or exclusive.
Recommended Free Tools
That is a validation recommendation, not a demonstrated customer-outcome improvement: the benchmark does not establish that this check improves real-world results. Its contribution is showing how a mismatch can look—correct amount and explanation, incorrect structured citation—and why prose review alone may miss it.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
