The answer reappeared because the application saved it to conversation history even though its check ended in an error, then sent that history to the model on the next call. The reported fix withholds answers whose checks did not complete, while keeping passing answers and previously accepted history.
What “remembered” means here
In this case, “remember” means the application retained an answer in conversation history and replayed it as context on a later model call. It does not mean the model learned the answer during training, and the experiment did not test a chat product’s long-term memory feature.
As an Amazon Associate I earn from qualifying purchases.
The report concerns OpenAI Agents JS pull request #1938, submitted by jbeckwith-oai and merged on September 15, 2026 UTC. The reported candidate revision was 457dfff8ce30d19ccbd4a3796ec482356dc19dfa; its control used the predecessor of runner/guardrails.ts from revision 8ac97dfedd0395beeb38edb0a17b83a9b89c3354. The report says that file was the pull request’s only changed production module.
Why an error could leave the answer available
A check that explicitly rejects an answer and a check that throws an error are not the same outcome. According to the report, the prior behavior already withheld answers after explicit rejection. The defect was that when a check threw—so it had not completed—the answer remained eligible for history and could be replayed as context.
#1 Best Overall
An error does not prove the answer is wrong. But it also does not establish that the answer passed. The reported fix treats an incomplete check as a separate state: withhold that answer from the next model call rather than treating it as accepted or claiming it was disproved.
What changed in the reported fix
The report says answers whose checks did not complete were withheld from subsequent model context. Passing answers and previously accepted history remained. This is an admission rule for future prompts, not a claim that every failed attempt must be erased from storage or that the underlying answer is false.
Rank #2
That distinction matters for debugging: a system can preserve an attempt for investigation while separately deciding what content is safe to present to a model as established context. The report proposes this separation as a design implication; it does not say the experiment implemented or tested a separate diagnostic record.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsWhat the controlled test found
The authors used a scripted model to return a distinctive marker while real library code handled conversation and storage. They observed stored items and the input to the next model call, across streaming and non-streaming runs and two session implementations: MemorySession and an append-only session. The report says no live model service or external database was used.
| Checker outcome | Old logic: marker replayed | Reported fix: marker replayed |
|---|---|---|
| All checks pass | 4/4 constructed cases | 4/4 constructed cases |
| Explicit rejection | 0/4 constructed cases | 0/4 constructed cases |
| A check throws | 4/4 constructed cases | 0/4 constructed cases |
| One check passes and another throws | 4/4 constructed cases | 0/4 constructed cases |
The 4/4 and 0/4 counts describe the constructed combinations of run mode and session type, not production incident rates or a general reliability measure. Across four checker outcomes, two run modes, two session types, and two code versions, the authors counted 32 observations. They separately report that 71 tests in the corresponding upstream test file passed, including additional tool-output behavior; that test count is not a reliability score.
What the result does—and doesn’t—prove
The experiment supports a narrow conclusion about the tested code paths: under the old logic, answers associated with throwing checks were replayed in the constructed cases; under the reported fix, they were not. The report does not establish correctness for every custom session store, nor does it establish a corresponding defect in CodeFlowMu.
Rank #4
A completed check is bounded by its configured requirements. Passing means the answer was accepted under those checks, not that it is necessarily true in the real world. Likewise, a check that throws leaves the answer unverified; it does not establish falsity.
Recommended Free Tools
How to investigate a similar replay
When a failed or interrupted answer appears later as if it were established, trace both storage and prompt construction. Ask: Does an answer marked as failed later reappear as an already-established fact?
Quick Recap
- Identify the exact checker outcome: pass, explicit rejection, or error. Do not collapse a thrown check into a rejection.
- Inspect what the session stored after that outcome, then inspect the actual input assembled for the next model call. Stored history and admitted prompt context are related but distinct questions.
- Check whether the behavior differs between streaming and non-streaming paths or between session implementations.
- If verification can finish later, associate its verdict with the precise answer, turn, and checking rules it evaluates. The report raises this as a design question; it did not test deferred verification or such an association mechanism.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
