Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →A numeric-answer grader can mark a correct arithmetic response wrong if it extracts the first number in the response rather than the answer. Erik Hill says that is what happened in his own grading harness: a model could repeat a number from the prompt before giving the correct result, and a regular expression compared the earlier number with the expected answer.
How the grader picked the wrong number
Hill describes a grader that uses a regular expression to find a number in a model’s output, then compares that match with the expected answer. Because the expression returns the first match, a response that restates the problem can place an input number ahead of its result.
As an Amazon Associate I earn from qualifying purchases.
For example, “Compute 3 + 4 * 5” has an expected answer of 23, but a response that repeats the expression begins with 3. Likewise, “What is 0.3 minus 0.1” expects 0.2, while a response that echoes the question begins with 0.3. In either case, the grader may compare the prompt’s number with the expected answer, not the arithmetic result later in the response.
That explains why a response can contain the correct answer and still fail: the issue is which number the grader extracts, not necessarily the model’s arithmetic.
#1 Best Overall
What Hill observed in his harness
Hill reports that seven of the eight tasks in his suite began with a number other than the expected answer. He says the float-subtraction task failed across nine models, and one model failed in 60 of 60 runs despite including the correct result in its response.
Those counts describe Hill’s harness and reported runs. They are not an independently replicated statistic or a general failure rate for models or numeric graders. Hill also notes that the other vulnerable tasks passed because the tested models answered tersely; that behavior does not make the first-match grader safe when a response restates the question.
How instruction wording changed the extracted number
Hill tested the same float-subtraction task with three instruction wordings across three models, for nine reported calls. With the as-shipped instruction, “Give the number to one decimal place,” the first extracted number was 0.3 and all three calls failed. With “Give only the number” and “Number only,” the extracted value was 0.2 and all three calls for each wording passed.
This small experiment suggests that wording affected whether a response restated the input in that test. It covered one task and three models, so it does not establish how broadly the effect generalizes. More importantly, making a prompt elicit terse answers does not change the grader’s first-match behavior.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Why the obvious fixes involve trade-offs
Hill does not settle on a grader fix, and the alternatives have different risks:
- Read the last number: this could capture the answer when a response shows its work, but it could also accept a response that ends with a wrong value.
- Require a bare number: this avoids ambiguity for the extractor, but it can turn an arithmetic task partly into a formatting task. A model may know the answer yet fail because it did not follow the output format.
Hill’s practical warning is: “If you grade numeric answers with a regex: check which end of the string you are reading, and check it against an output that restates the question.” — Erik Hill, article posted September 30, 2026.
Quick Recap
Best Value
- Full of different activities to help your child develop their skills
- Contains one sixty-four page workbook
- Available in a variety of different age groups
- Available in different themed activity books
- Made in USA
What to check when a correct answer is marked wrong
- Inspect the actual response passed to the grader, including any repeated numbers from the prompt.
- Check whether the numeric extractor returns the first match, the last match, or something else.
- Compare the extracted value with the expected answer before treating the failure as an arithmetic error.
- Test responses that restate the problem as well as responses that provide only a number; passing terse outputs alone may not expose a first-match bug.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
