The same benchmark runs produced three token totals: 109,079 tokens for an uncut conversation transcript, 6,671 for the assembled memory context, and 9,908 billed input tokens for a hand-picked subset of 27 questions. They describe different things, so they cannot be treated as interchangeable measures of savings. The practical lesson is to ask what was counted—and whether the number says anything about answer quality or total task cost.
What the three token totals count
In a September 28, 2026 post on DEV Community, the account “belcore” reported figures from benchmark runs conducted in August 2026. The setup used LongMemEval_S, a 500-question dataset with about 109,000 tokens of prior conversation per question, or roughly 500 turns. The answer model was GPT-5; the author says the harness was frozen, a fixed seed was used, and context counts used the o200k_base tokenizer.
As an Amazon Associate I earn from qualifying purchases.
| Reported figure | What it counts | Scope |
|---|---|---|
| 109,079 tokens per call | The full prior transcript, uncut | Context count |
| 6,671 tokens per call | The assembled memory context | Context count |
| 9,908 billed input tokens per call | Input including the prompt and question | Hand-picked subset of 27 questions |
The author reported a 93.9% reduction comparing the two context counts, and about a 91% reduction in billed input for the 27-question subset. Those percentages answer different questions. The first compares transcript context with assembled memory context; the second compares billed input, including prompt and question, on a selected subset. Neither is a universal savings rate.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11The post’s useful framing is: “How large the saving looks depends on what you compare, so here are the same runs counted three ways.” The figures and conclusions are the account author’s report, not an independently verified result.
#1 Best Overall
Why “tokens saved” needs a scope
A token total can refer to the conversation supplied as context, a selected or assembled memory, or the full input billed for a request. These are not equivalent. A comparison that counts only context may omit tokens for the prompt and question; a billed-input figure that includes them has a broader scope. A percentage is meaningful only when its numerator and baseline use matching definitions.
- Context reduction: compare the same conversation’s full context with the context actually assembled for the model.
- Billed-input reduction: include the prompt and question on both sides of the comparison, and state how many requests were measured.
- Task efficiency: account for answer quality, retries, fixes, and latency as well as tokens.
For this post, the author reports context reductions and a subset-based billed-input reduction. It does not establish a production-wide saving or a cost per successful task.
What the benchmark does—and does not—establish
Sample selection limits the billed-input result
The 9,908-token billed-input figure and roughly 91% reduction come from 27 hand-picked questions. The author describes that result as indicative only. It should not be presented as the measured result for all 500 LongMemEval_S questions or as a forecast for production traffic.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsToken counts do not measure answer quality
The post says accuracy is a separate metric and does not report an answer-quality result for the token comparison. A smaller context is not, by itself, evidence that answers stayed equally accurate.
End-to-end cost was not measured
The author did not measure cost per successful task, including retries and fixes. A lower input count alone therefore cannot show whether the full workflow costs less to complete a task.
Audit observations call for caution
The author reports that 38% of audited wrong answers involved a gold label judged wrong or defensible either way, and notes that n = 499 cannot resolve small effects. These are the author’s audit observations and stated limitation; they are not independent verification of label quality or benchmark performance.
What the multi-pass retrieval experiment adds
A secondary experiment tested a multi-pass approach that re-checks and retrieves evidence before answering. Its results differed sharply by sample design:
Recommended Free Tools
| Sample | Reported result | Interpretation |
|---|---|---|
| 27 hand-picked questions with missing evidence | Correct answers rose from 16 to 20 | A selected set; not a general estimate |
| Random sample of 103 | Attributable effect of +2, within a ±5.2 noise bar | The result does not establish a clear improvement |
On the hand-picked subset, the author reports about 2.9 times the input tokens and 2.6 times the latency. One case gained extra evidence but changed a correct aggregation answer into a wrong one. The author says this approach is not being shipped as an improvement. The selected-set change from 16 to 20 should not be conflated with the random-sample result.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to evaluate a token-savings claim
Before using a percentage to choose a memory or context strategy, check that the comparison answers the same questions on both sides:
- Define the count: is it transcript context, assembled context, or billed input?
- Check inclusions: are the prompt and question counted?
- Check the sample: how many questions were used, and were they hand-picked or randomly sampled?
- Look for answer quality: does the report show whether accuracy or task success changed?
- Count the whole task: are retries and fixes included in cost per successful task?
- Compare latency: does the approach add passes or time even when it reduces context?
For the September 2026 post, the token scopes and sample caveat are reported, but answer quality for the core comparison and end-to-end task cost are not. Those unanswered measures matter if the goal is choosing an approach for real workloads rather than describing a token count.
Source and disclosure
The figures and experiments above are from DEV Community’s “Token savings depend on what you count: three numbers from the same benchmark runs”, posted September 28, 2026. The post discloses that its author is affiliated with Belcore, a memory and context layer for LLM apps. That disclosure is relevant context for interpreting the author’s claims; it does not establish any particular commercial terms.
Free tools Windows power users keep installed
One-click scans. No signup required.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
