Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsIn one vendor-run benchmark, a model given a prepared context pack used 98.77% fewer input tokens than the full-context version and scored higher on strict accuracy: 100% observed versus 78.79%. That is a notable result, but it does not show that cutting tokens alone improves accuracy. The test also changed how evidence was selected and prepared, and it has not been independently reproduced.
What the 98.77% figure measures
Josh Hodgetts, founder of Jylus, reports the result from a frozen benchmark of 528 questions. The percentage compares average input tokens for the Jylus Context Pack with the full-context baseline—not output length, total system cost, or a general property of large language models. The same Gemini 3.1 Flash Lite settings, questions, and deterministic scorer were used across the three evidence paths, according to the October 5, 2026 DEV article.
As an Amazon Associate I earn from qualifying purchases.
| Evidence path | Average input tokens | Strict accuracy |
|---|---|---|
| Full context | 205,129 | 78.79% |
| BM25-only RAG | 8,128 | 76.14% |
| Jylus Context Pack | 2,532 | 100% observed (528/528) |
All figures in the table are vendor-reported results for this benchmark, not independently verified measurements. The token reduction is the difference between the reported averages of 205,129 and 2,532 tokens.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Why fewer tokens did not explain the accuracy result
The comparison changed more than context size. Full context supplied the broad source material; BM25-only retrieval selected records using a lexical retrieval method; and Jylus says its Context Pack retrieves and prepares evidence, resolves state and relationships, and includes source references, conflicts, or gaps. That means the benchmark cannot isolate whether the higher score came from the smaller input, the evidence preparation, or their combination.
#1 Best Overall
The BM25-only result is an important check against a simplistic conclusion: it used substantially fewer input tokens than full context but scored 76.14%, below full context’s 78.79%. This is evidence about one BM25-only implementation on this benchmark, not a verdict on every retrieval-augmented generation system.
What the benchmark can—and cannot—establish
The vendor’s benchmark methodology page describes 528 unique adversarial questions across four data domains and 11 adversarial classes, with 1,584 completed evaluations. It says the same model settings and scorer were used for each path and that raw answers, provider request IDs, timings, hashes, and checkpoints were retained. The page also says the compiler’s ranking, fusion, evidence selection, temporal and relationship processing, optimization, and tuning are proprietary.
- “100% observed” is bounded: Hodgetts defines it as 528 of 528 questions passing the benchmark’s strict scoring. It is not a guarantee of universal accuracy or performance on other tasks.
- Independent reproduction is not established: both the article and methodology page say the benchmark has not been independently reproduced.
- More reported metrics are still vendor results: the methodology page includes latency, cost, recall, and hallucination-trial measures, but they do not provide independent corroboration.
- The result is workload-specific: a frozen set of adversarial questions can show how the systems performed on that set; it cannot by itself predict performance on a different application’s records or questions.
Why temporal records are a hard test for AI context
The motivating problem is not merely finding a passage with matching words. Records can conflict across time: an old price may be replaced, while a correction entered later may apply to an earlier event. An answer to “What do we know now?” may differ from an answer to “What was known then?” A useful evidence pipeline has to preserve that distinction rather than flattening all records into a single current-looking summary.
For support histories, changing subscriptions, or incident investigations, the practical questions include which source supports a fact, whether it applies to the requested time, whether a later record superseded it, whether sources disagree, and whether there is enough evidence to answer. A smaller prompt helps only if the selection process retains the records and qualifications needed to answer those questions faithfully.
How to assess a context-preparation approach
The benchmark suggests a useful evaluation framework for teams considering retrieval or context compression. Test the evidence path on representative questions, especially cases with stale facts, corrections, conflicting sources, and missing evidence. Compare both the answer and the provenance supplied to the model; token count alone is not an adequate quality measure.
- Freeze the workload and scoring rules. Use the same questions, model configuration, and deterministic scoring across candidate evidence paths so the comparison is interpretable.
- Track input size and answer quality together. Record average input tokens alongside strict accuracy; also inspect failures rather than treating an aggregate score as the whole story.
- Check temporal scope and provenance. Verify that the selected context makes clear which record supports a claim, when it applies, and whether a later entry changes its status.
- Include disagreement and uncertainty cases. Measure whether the system preserves conflicts and gaps instead of silently choosing a source or inventing a resolution.
- Seek reproducibility. Retain evaluation artifacts and document the evidence-selection method sufficiently for another team to inspect or independently repeat the test.
What to take from the result
The Jylus benchmark reports a striking combination: much lower model input and perfect observed strict accuracy on its 528-question test. It is best read as a case for evaluating evidence preparation—not as proof that shorter prompts are inherently more accurate. The decisive question for another application is whether its own smallest sufficient context preserves the right facts, time boundaries, sources, conflicts, and unknowns.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
