Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Yes. A tool call can use a real parameter, contain an allowed value and pass schema validation while still contradicting what the user asked for. In a 2026 benchmark, Bowen Rui tested this semantic failure with requests such as “Thai dinner recipes that take 30 minutes or less and have no peanuts in them. My son is allergic.” An inclusion-only filter cannot express “exclude peanuts”; putting “peanuts” into it reverses the request.
What a validator can—and cannot—tell you
A schema validator can check whether a call has the expected structure, uses defined parameter names and supplies values that meet the schema’s rules. Those checks matter, but they do not establish that the call preserves the user’s meaning. A call can be valid in form and wrong in effect.
![Allergy-Free and Easy Cooking: 30-Minute Meals without Gluten, Wheat, Dairy, Eggs, Soy, Peanuts, Tree Nuts, Fish, Shellfish, and Sesame [A Cookbook]](https://m.media-amazon.com/images/I/51iTyrCrzaL._SL160_.jpg)
As an Amazon Associate I earn from qualifying purchases.
Consider a tool with an include_ingredients parameter but no exclusion parameter. A request for recipes with no peanuts cannot safely be represented by adding “peanuts” to the inclusion list. The parameter means to include an ingredient, not filter it out. The resulting call may be syntactically acceptable while asking the tool for the opposite of what the user intended.
Recommended Free Tools
That distinction is especially important when a mistaken call could have serious consequences. Rui uses an allergy-related request to illustrate the stakes; the benchmark does not test food safety or establish that any recipe result is safe for someone with an allergy.
#1 Best Overall
What Rui’s benchmark tested
Rui’s benchmark contains 202 items involving 75 invented tools, grouped into eight families: ordinary missing-parameter requests, near misses involving differently named equivalent parameters, requests the tool cannot express, pressure to use enum values, nested-field errors, attempts to repurpose an existing parameter, controls and matched controls.
For each item, the model produced a tool call as JSON text. Rui scored the response with schema validation and a prewritten, item-specific check. The repurposing items form the central test: would a model put the user’s request into a real parameter whose meaning is different? Matched controls help distinguish inappropriate parameter use from parameter use that is actually suitable.
The benchmark compared three prompt conditions:
neutral: asks for exactly one tool call.instructed: adds an instruction to use only parameters defined in the tool schema.may_decline: permits a one-sentencecannot_doresponse.
The Kaggle comparison covered eleven models across all 202 items and all three conditions, using temperature zero and one sample per item. Rui also describes earlier local pilot and validation runs on Nebius Token Factory. Those local runs are separate from the Kaggle comparison; their results should not be combined.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteHow often did models repurpose a valid parameter?
In Rui’s Kaggle neutral condition, 46 of 308 replies on the 28 repurposing items were flagged as repurposed calls: 14.9% pooled across eleven models. The count per model ranged from zero to thirteen. This is an author-reported result for these benchmark items and runs, not a general error rate for deployed agents.
When the prompt allowed a model to decline, Rui reports 21 repurposed replies out of 308, or 6.8%; eight of the eleven models repurposed nothing under that condition. That improvement came with a trade-off: the condition also produced many declines on requests where the tool could have helped at least partly.
Rui’s local validation results show a similar direction but different rates: repurposing fell from 92 of 308 replies (29.9%) when a call was required to 49 of 308 (15.9%) when declining was permitted. These are local results from a different run, not another part of the Kaggle total. Rui says the local repurposing detector was revised after inspecting replies, which is another reason to keep the sets distinct.
Rank #3
- Used Book in Good Condition
The schema-only instruction had little effect in the local validation run, where the count changed from 92 to 91. In Kaggle, the count was 46 in the neutral condition and 34 in the instructed condition. That pattern fits the specific failure being tested: the misleading parameter already exists in the schema, so merely telling a model to stay within the schema does not resolve whether its use matches the request.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why field names and allowed values are not enough
Rui reports several examples of a valid-looking field being used for a different concept:
authorused to represent “reviewed by.”older_than_daysused for “modified within the last seven days.”- A canceled status used for subscriptions “on pause.”
ccused when the user asked for a blind copy.
In each case, checking that the field exists—or that a status is an allowed enum value—does not show that the value means what the user asked. A validator can reject malformed calls and catch many unsupported values; semantic alignment needs a separate check against the request and the tool’s actual parameter meanings.
For the peanut example, the issue was not just that an inclusion filter cannot express an exclusion. In the Kaggle neutral condition, Rui reports that gpt-5.4-mini sent include_ingredients: ["no peanuts"]. Rui characterizes this as an inclusion filter receiving the negated phrase, rather than an exclusion request. In the local validation run, the peanut example appeared in five of 33 replies. These figures describe the author’s runs, not the prevalence of this exact error elsewhere.
What the results say about allowing a decline
Giving a model an explicit way to say it cannot carry out a request reduced repurposing in both reported sets of runs. That is useful when the alternative is a call that looks valid but reverses intent. It is not a complete solution: a model may decline even when the tool could return a broader or partial result that a caller could filter afterward.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rui notes that a decline could be scored as correct even when such a broader result might have been acceptable. The evaluation checks whether a decline is present, not whether its explanation is accurate. So the reported decline figures measure the benchmark’s defined outcomes; they do not establish that every refusal was necessary or correctly explained.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What this benchmark does not establish
- It is not a test of native tool-calling APIs. The experiment represents tool definitions and calls as text and JSON text. Rui says the results do not establish how native tool-calling interfaces behave.
- It does not establish a general rate across deployed agents. The percentages come from a particular benchmark, prompt design, model sample and scoring method.
- It does not show that reasoning causes lower repurposing. Rui cautions that reasoning and non-reasoning groups contain different models, so the comparison cannot isolate reasoning as the cause.
- Small differences do not support confident rankings. Rui warns against ranking models that differ by only one or two items.
- The per-item checks are narrow. A written check can assess the specific intended behavior encoded for an item; it is not a universal measure of semantic correctness.
Rui reports 6,666 Kaggle calls across eleven models, 202 items and three conditions. That scale gives the comparison structure, but it does not remove the limits of the item set or scoring approach. The article describes public Kaggle and GitHub resources, but no independent replication or outside validation is established in the material presented.
What to take away when evaluating tool calls
Treat schema validation as a necessary structural check, not proof that a call is safe or faithful to the request. A robust evaluation should separately measure whether the call’s meaning matches the user’s intent, whether an unsupported request is handled appropriately, and whether a model declines unnecessarily when partial assistance is possible. Rui’s benchmark makes the gap concrete: a valid parameter can still produce the wrong action when its meaning is repurposed.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
