If you tell an assistant, “Remember this next week,” the key question is not how much text it can handle in one sitting. It is whether the system saves the information outside this request and brings it back when needed. A context window is the finite token budget for a particular model request; memory usually means information stored beyond that request and selectively supplied later. The model can use only what reaches its active context on the call it is answering.
What is a context window?
A context window is the amount of text and other request material a model can work with for a single call. Anthropic defines it as the text a model can reference while generating a response, including the response itself. It is a per-request working space, not a durable memory bank.
As an Amazon Associate I earn from qualifying purchases.
The budget may include more than the latest user message. Depending on the system and model, it can include system and developer instructions, earlier conversation turns, tool definitions and results, retrieved passages, and generated output. Some models also count reasoning tokens. The exact accounting depends on the API or product. OpenAI’s conversation-state documentation describes how conversation material can accumulate across requests; Anthropic’s context-window documentation explains what can consume the window.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →If a request exceeds the available budget, the system may reject it, truncate material, or manage the conversation by omitting or compacting content. A larger window can reduce the need to split material, but it does not ensure that every detail will be recalled reliably. Anthropic cautions that accuracy and recall can degrade as token counts grow.
#1 Best Overall
What does “memory” mean?
Memory is not one standardized model feature. It can mean a product’s saved chat information, application-managed files, a generated summary, retrieved passages from a knowledge base, or simply old conversation turns that an application continues to pass in. In each case, persistence happens outside the model’s active request; the relevant information must be made available to the model again when it is needed.
For example, Anthropic’s API memory tool lets Claude operate on files in a memory directory that persists across sessions. The application runs those operations and determines where the data is stored. At a later step, relevant content can be retrieved and added to the active request. This is a persistence-and-retrieval mechanism, not an expansion of the context window. See Anthropic’s memory-tool documentation.
Some consumer chat products also offer saved memories or search across past chats. Those features are product-specific: how information is selected, retained, and controlled depends on the product and account. They should not be assumed to work the same way in every chatbot. Anthropic describes its own user-facing approach in its Claude chat search and memory help page.
How summaries, compaction, and RAG fit
Conversation history and compaction
The simplest way to maintain continuity is to send earlier turns again with each request. This keeps the relevant history in the current context, but the prompt grows as the conversation continues. A system can instead summarize or compact older turns, retaining a shorter representation and adding it to a later request. OpenAI’s conversation-state guide discusses managing growing conversation state, including compaction. A summary saves space but can omit details, exact wording, or distinctions that later prove important.
Retrieval-augmented generation
Retrieval-augmented generation (RAG) is useful when the source material is too large to include all at once, or when only a subset is relevant to a question. A typical flow is:
- Store: Keep documents or other material in an external corpus.
- Retrieve and select: Search for passages relevant to the current question.
- Add to the request: Place selected passages in the model’s active context.
- Generate: Ask the model to answer using the material it received.
The model does not thereby receive the entire corpus. It sees the selected content added to this particular request. Anthropic’s RAG explanation describes combining lexical search, such as BM25, with semantic embeddings and ranking to select passages. Retrieval can still miss relevant material or return an incomplete set; a successful simple “needle in a haystack” test does not establish reliable performance on multi-document questions. Google’s long-context documentation cautions that multi-item retrieval may be less accurate than finding one isolated item.
Context caching
Context caching is an optimization for repeated long inputs, not a separate kind of memory that makes information automatically available to every future request. Google documents caching for repeated context in its long-context guide. Whether it saves time or cost depends on the model, request pattern, and current pricing; caching does not remove the need to decide what content belongs in a call.
Which approach should you use?
| Approach | Persists between requests? | What the model gets now | Main trade-off |
|---|---|---|---|
| Pass conversation history | Only if the application retains and resends it | The history included in this call, subject to the context budget | Continuity is straightforward, but the request grows and may need trimming or compaction |
| Summarize or compact | Yes, if the system saves the summary | A compressed account of earlier material | Uses fewer tokens, but details can be lost or distorted in compression |
| Application-managed memory | Yes, in storage controlled by the application | Only the stored information selected and added for this call | Supports cross-session continuity, but requires choices about what to save, retrieve, and retain |
| RAG over a corpus | The corpus persists outside the call | Retrieved passages selected for the current question | Can address large collections selectively, but depends on retrieval quality and adds system complexity |
| Larger context window | No, not by itself | More material can fit in one call, subject to that model’s limits | Can reduce splitting, but does not guarantee reliable recall or preserve information across sessions |
These options solve related but different problems. A larger window is useful when the needed material fits and benefits from being considered together. Retrieval suits a large corpus or selective lookup. Explicit stored state or memory is appropriate when facts should survive across sessions. Many applications combine approaches—for example, storing conversation state while retrieving a few relevant documents for each question.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What developers should evaluate
For a chat, assistant, or agent system, the important design question is not simply “How large is the context window?” It is how the system selects, persists, and supplies the information needed for a particular task. Evaluate the whole path from storage to answer, including:
- Persistence: Which information must survive a request, a session, or a longer period?
- Selection and recall: How does the system choose what goes into the prompt, and how does it behave when relevant material is missing?
- Detail preservation: Could a summary erase a specific constraint, or could retrieval overlook a passage needed to answer correctly?
- Token use, latency, and cost: History, retrieved passages, and tool material all add work; the balance depends on the model and implementation.
- Privacy and control: Where is state stored, how long is it retained, and who can inspect, change, or delete it?
- Engineering effort: Passing history is simpler than operating a retrieval system, while memory and RAG require deliberate storage and selection logic.
Test with realistic multi-step questions, not just a single fact placed in a long prompt. Include follow-up questions that depend on details from different parts of the conversation or corpus, and check whether the right details were supplied to the model before judging its answer. Model capacities and product behavior change; for instance, Anthropic’s documentation lists 1-million-token windows for some named API model versions and 200,000-token windows for others, while Google says many Gemini models support windows of 1 million tokens or more. These are vendor specifications for particular products and versions, not a universal measure of dependable memory.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
