The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Yes—but only for one narrow guarantee. An enum-style grammar can stop a language model from emitting a FAR or DFARS clause number that is absent from an approved registry. It cannot by itself determine whether that valid number answers the user’s question. In Raihan Sikder’s corrected 2026 test, adding a NONE option created the opposite problem: the model refused every prompt that named a clause number, including all 15 real clauses.
What the experiment actually tested
The question was not whether constrained decoding can make a model understand federal acquisition regulations. It was narrower: can a grammar prevent fabricated FAR/DFARS identifiers, and can an abstention option make the model decline questions for which no registered clause is appropriate?
As an Amazon Associate I earn from qualifying purchases.
Raihan Sikder compiled a FAR/DFARS clause registry into an enum-style decoding grammar and tested it with Qwen2.5-1.5B-Instruct. The corrected version used 75 prompts, five decoding conditions, and greedy decoding. The prompt set contained four groups:
- 30 deliberately fake topics, such as subjects that should not map to a single clause;
- 15 prompts naming real clause numbers;
- 15 prompts naming numbers absent from the registry; and
- 15 obscure but real topics.
The registry had 1,128 raw entries that collapsed to 1,056 distinct canonical clause IDs. That distinction matters: duplicate or differently formatted source entries do not create additional identifiers. The grammar guaranteed that a selected ID belonged to those 1,056 canonical values. It did not guarantee that the ID was relevant, that its title was genuine, or that the model had understood the regulatory issue.
How an enum grammar constrains a model
Free generation
With ordinary decoding, the model predicts the next token from its vocabulary. A clause-like string such as 52.227-7 may look plausible even when it is not in the registry. The decoder has no built-in concept of an approved identifier list unless the application supplies one.
Registry-constrained generation
An enum grammar changes the legal output space. At each decoding step, the grammar masks tokens that cannot lead to one of the registered strings. The final answer must therefore be one of the allowed IDs. This is a syntactic or membership guarantee, not a legal interpretation.
In this test, the grammar could prevent an invented number but could still select a real number for the wrong reason. It also could not independently validate accompanying prose. A model might emit a registered ID and then attach a description that was never present in the source material.
The registry boundary
The guarantee ends at the registry boundary. If the registry is stale, incomplete, incorrectly normalized, or missing an applicable clause, the grammar faithfully constrains the model to a defective set. FAR and DFARS identifiers and their associated regulations change over time, so an operational system needs a process for refreshing and approving the registry rather than treating the enum as permanent truth.
Results: fabrication fell to zero, but abstention did not discriminate
| Condition or measurement | Reported result | What it establishes |
|---|---|---|
| Free generation on 60 prompts with no single correct clause | 57 of 60 fabricated a clause number | Unconstrained output frequently produced plausible-looking identifiers when the prompt did not have one clear answer. |
| Enum grammar on the same 60 prompts | 0 of 60 fabricated a clause number | The grammar enforced membership in the registry. |
Enum grammar plus NONE, all 60 judged prompts |
62% overall good-answer rate | The combined system often avoided bad selections, but the score includes abstentions and does not prove clause understanding. |
Fake-topic prompts under grammar plus NONE |
22 of 30 abstained (reported as 73%) | Abstention was common when the topic was intentionally implausible. |
Real-clause prompts under grammar plus NONE |
15 of 15 answered NONE |
The abstention branch rejected every named real clause in this group. |
Absent-number prompts under grammar plus NONE |
15 of 15 answered NONE |
It handled missing identifiers correctly, but the same output occurred for real identifiers. |
| Constrained-decoding speed | About 10% token-speed overhead in the first run | There is a measurable runtime cost to enforcing the grammar. |
The 15 real-clause prompts and 15 absent-number prompts were all prompts that named a clause number. Because the system returned NONE for all 30, its perfect absent-number result cannot be interpreted as evidence that it learned to distinguish real from missing IDs. It may simply have learned a blanket refusal pattern.
Why a valid identifier can still be a wrong answer
Constrained decoding controls the form of an answer, not its meaning. A model can choose a member of the registry that is unrelated to the user’s topic. Sikder recorded an example in which the model emitted 52.227-7 and paired it with the invented title “Quantum Cryptography Requirements for Field Radios.” The number was registry-valid; the title was not grounded by that validity.
This creates at least three separate error classes:
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitches- Invalid identifier: the emitted number does not exist in the approved registry. An enum grammar can block this class.
- Irrelevant valid identifier: the number exists but does not address the question. A grammar alone cannot block it.
- Ungrounded fields: the ID is valid, but the title, applicability statement, or explanation is invented or mismatched. Output-field validation or source retrieval is required.
That is why the concise description of the result is: “Constrained decoding buys guarantees, not accuracy.” The guarantee is valuable when an invalid identifier would cause a downstream system to apply the wrong rule, but it should not be advertised as regulatory reasoning.
What the 62% score does—and does not—mean
The 62% good-answer rate for grammar plus NONE is a result for this 60-prompt judged set, not a universal accuracy estimate. Correct abstentions on fake or absent cases contribute substantially to that number. On prompts that named real clauses, the evaluation mostly rewarded copying a number already written in the prompt. The title associated with that number was not fully grounded or graded, and free generation also copied all 15 named real numbers. Therefore the grammar did not demonstrate deeper understanding on that part of the test.
The denominator also needs care. The corrected experiment contained 75 prompts across four groups, while the headline fabrication and good-answer figures are reported for 60 judged prompts. Those figures should not be silently generalized to all 75 prompts or to another model, prompt style, or regulatory corpus.
Why the NONE branch failed
Abstention is a classification problem
Adding NONE gives the decoder an extra legal output, but it does not teach the model when that output is appropriate. If the prompt format, demonstrations, or scoring reward caution, the model may discover that refusing is safer than selecting a clause. In this test, that strategy produced 15 false refusals on real clauses while still producing 15 correct refusals on absent numbers.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Balanced cases are essential
A useful abstention evaluation must contain matched answerable and unanswerable examples. For every class of missing or irrelevant topic, include real clauses with similar wording and difficulty. Report both:
- Abstention precision: among responses of
NONE, how many truly required refusal? - Abstention recall: among prompts that truly required refusal, how many received
NONE?
Reporting only the number of refusals can make a model that refuses everything look safe. The real-clause and absent-number groups in this test expose that problem directly.
A safer implementation pattern
Use the grammar as one layer in a pipeline rather than as the whole decision system.
- Maintain a canonical registry. Normalize clause strings, remove duplicates, record effective dates and source provenance, and preserve the 1,056 distinct IDs separately from the 1,128 raw entries that produced them.
- Constrain identifier output. Apply an enum grammar when the hard requirement is “never emit an identifier outside this approved set.” Keep the grammar tied to the exact registry version used for the request.
- Retrieve authoritative text. After an ID is selected, fetch the official clause text, title, date, and applicability metadata from the controlled corpus. Do not let the model generate those fields from memory.
- Match the question semantically. Compare the user’s topic with the retrieved clause scope, contract context, thresholds, and required conditions. A valid ID that fails this check should be rejected or sent for review.
- Validate every output field. Check that the title and any quoted language come from the retrieved record. Treat a valid number with an unverified title as a failed answer, not a partial success.
- Handle abstention explicitly. Define when
NONEis allowed, provide balanced examples, and route uncertain cases to a human or a second verification pass rather than silently treating refusal as correctness. - Log the decision. Store the registry version, retrieved record, model output, validation result, and reason for abstention. Regulatory answers need an audit trail that a bare token cannot provide.
What to measure in a production comparison
A system that claims to improve FAR/DFARS clause selection should report more than one accuracy percentage. At minimum, compare these axes:
Free tools Windows power users keep installed
One-click scans. No signup required.
| Metric | Question it answers | Why it matters |
|---|---|---|
| Invalid-identifier rate | How often is an ID outside the approved registry emitted? | This is the direct benefit a registry grammar is designed to provide. |
| Relevance or semantic accuracy | Does the selected clause actually address the user’s topic? | Membership does not establish applicability. |
| Abstention precision and recall | Does NONE appear for the right cases? |
Separates calibrated refusal from blanket refusal. |
| Title and field grounding | Are titles, descriptions, and quotations copied from authoritative text? | Prevents a valid ID from carrying fabricated metadata. |
| Robustness | Do results hold across model sizes, prompt templates, and topic difficulty? | The reported test used one small model and one template per prompt kind. |
| Runtime overhead | How much speed or compute is lost? | Constrained decoding had about a 10% token-speed cost in the first run. |
Limits of the reported case study
The evaluation used one 1.5B-parameter model, 75 prompts, greedy decoding, no fine-tuning, one prompt template for each kind, deliberately absurd fake topics, and one abstention wording. Those choices make the result easy to interpret, but they also limit transferability. A larger model, a different grammar engine, realistic acquisition questions, sampled decoding, or carefully written abstention instructions could change the rates.
Best Value
The test also did not fully ground or grade generated titles. That omission is important because a user may rely on the title more than the identifier. A future benchmark should score the identifier, title, applicability, citations, and refusal decision as separate fields instead of collapsing them into one good-answer label.
No independent replication or broader benchmark is established here. Treat the figures as a small corrected 2026 case study, not as an estimate of performance across FAR/DFARS systems. The registry and regulations are volatile; verify the current text and identifiers before operational use.
When a grammar is the right tool
- Use an enum grammar when an invalid identifier would be unacceptable and the approved set is known and maintained.
- Add retrieval and field-level verification when the answer must include an authoritative title, quotation, or applicability explanation.
- Add semantic matching when users ask by topic rather than by a known clause number.
- Use a separately evaluated abstention classifier or review queue when refusing an answer has material cost.
- Do not treat zero fabricated IDs as proof that the model understands procurement regulations.
The practical verdict is narrow but useful: grammar-constrained decoding solved the identifier-format problem in this test, eliminating 57 fabricated numbers in the measured 60-prompt comparison. It did not solve relevance, grounding, or calibrated refusal. A dependable FAR/DFARS assistant therefore needs constrained output plus authoritative retrieval, semantic verification, and balanced abstention testing.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
