Free tools Windows power users keep installed
One-click scans. No signup required.
Usually it can state the new fact and still act on the old one. In Harsh Singh’s Stale Facts benchmark, models answered direct questions about a user’s changed details almost perfectly, but many fell back on the superseded fact when a request quietly assumed it. A model that can say your current employer is not necessarily a model that will draft your email to the right person.
What the benchmark tested
Singh’s write-up, submitted to the DEV Community as part of the Kaggle Benchmarking Challenge, separates two skills that most personal-assistant demos blur together: knowing a fact, and applying it when the fact is only implied by a request. The benchmark, called Stale Facts, is built from 34 conversation histories. In each one, a personal fact changes partway through a series of dated conversations, and the model is then asked questions about that fact.
As an Amazon Associate I earn from qualifying purchases.
The histories ran five to eight dated conversations and often wandered into unrelated subjects. The changing detail could be stated explicitly, implied by context, or buried among decoy details that looked similar but had not changed. Each history was followed by probes in five categories:
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitches- CURRENT: what is true now? For example, which company the user works for today.
- HISTORICAL: what was true at a past date? Singh’s example is “Which company was I working for in February 2026?”, where the correct answer is the earlier employer.
- PRESUPPOSED: a request that assumes the old fact without stating it, such as asking for cafes near a former home or drafting leave notes addressed to a manager who has since been replaced.
- ABSTAIN: the change appears only as a rumour, so a careful model should not treat it as settled.
- CONTROL: a nearby fact did not change, or a planned change was cancelled, so the model should keep the original value.
The full history was kept in the model’s context window on purpose. This isolates a narrower question than most memory products face: whether a model uses information it already has. It does not test whether a retrieval system can locate that information in the first place.
Why the categories matter
A single accuracy number hides the failure. A model can score well on CURRENT and HISTORICAL questions, which ask it to report a fact, and still do poorly on PRESUPPOSED requests, which require it to notice that a fact has changed and reason from the new value. The write-up’s central observation is that these are different tasks, and that the gap between them was large for some models.
Writing tasks were the hardest part of the PRESUPPOSED set, according to Singh. An email requesting leave, or an out-of-office note for a team the user had already left, requires the model to carry the change into the text it produces, not just into an answer to a question.
Rank #2
- Multilingual Handwriting Recognition Write naturally with the wireless writing pad instead of typing. Accurately recognizes handwritten Traditional Chinese, Simplified Chinese, English, Japanese, numbers, symbols, and mixed-language input for seamless text entry.
- Write Smarter with AI Boost your productivity with the built-in AI Writing Assistant. Draft emails, rewrite content, summarize documents, translate text, and generate ideas faster with the help of AI.
- Personalized Digital Signature Sign PDF documents, forms, contracts, and emails with your own handwritten signature, giving your digital documents a more professional and personal touch.
- Handwriting input to MS Word, MS PowerPoint, Google Docs, WeChat, Whatsapp, Line, and more. Win/Mac supported
- Plug & Play Wireless Convenience Simply connect the included wireless USB receiver and start using immediately—no driver installation required. Compatible with Windows and macOS for effortless setup.
Reported results
Singh reports testing 11 models from seven labs. Gemini 3.7 Flash also appears in the results as an additional model. Model names below are as written in the source; this article has not independently verified their versions or availability.
Singh reports that every model answered at least 18 of 20 CURRENT questions correctly. Stale-premise performance was far less uniform. The table below lists the models whose stale-premise scores are named in the write-up.
| Model (as named in the write-up) | CURRENT | PRESUPPOSED (stale-premise) | Other categories |
|---|---|---|---|
| GPT-5.4 mini | At least 18/20 (the floor reported for all models) | 0/20 | Not stated |
| GPT-6 Astra | Not stated | 18/20 | Not stated |
| Several other models | Not stated | 20/20 | The write-up reports perfect results across all listed categories for some models; individual names are not given in the source text reviewed here. |
These are one author’s scores on one 34-history benchmark. They describe behaviour on this test and should not be read as a ranking of those models for everyday use.
Where the failures cluster
The most useful pattern in the write-up is not a leaderboard but the way the change was communicated. Pooled stale-premise accuracy across the models was:
Rank #4
| How the fact change was given | Pooled stale-premise accuracy (author-reported) |
|---|---|
| Implied by context | 56% |
| Stated outright | 76% |
| Given as an explicit correction | 75% |
The implied case is the one that resembles ordinary conversation: people rarely announce every update, they mention it in passing. A model that treats an unannounced change as background noise will carry the old value forward, which is exactly the failure the benchmark is designed to expose.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Reading the result correctly
Singh’s summary is direct: “Knowing the fact and acting on it are two different skills.” The benchmark supports that claim for the models and histories tested. It does not establish a general rate of failure across real users, longer conversations, or the model versions people use today.
Limitations
- Sample size. With 34 histories and 84 probes, small differences between models may be noise. The results are best treated as exploratory.
- No retrieval or deployed memory. Full transcripts were in context. The test cannot say whether a vector store, a summary-based memory, or a commercial memory feature would produce the same outcomes. Singh proposes testing real memory systems and prompt-level interventions as future work.
- Authoring. Histories were drafted with LLM assistance from a detailed specification and reviewed individually. Singh reports that one broken item was caught and fixed.
- Grading. Singh describes using three judge models for ambiguous cases and manually reviewing a sample of judge verdicts. These are the author’s own procedures, not an independent audit.
What to check in your own assistant
The benchmark suggests a simple check that does not require any tooling. Update a personal detail in conversation, then ask for something that depends on the old value without naming it, for instance a message to a former colleague or a list of services near your previous address. Then see whether the reply uses the new detail. If it does not, the assistant may be reporting the update while still acting on the old one, which is the gap the Stale Facts results describe.
Use the category breakdown, not one headline score, to judge any model you evaluate. A model that answers “where do I work now?” correctly has shown only the first skill.
Harsh Singh’s Stale Facts write-up is the primary source for these figures. The benchmark’s own code and histories are referenced there rather than reproduced here.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
