Manage an agent’s context window as a per-request budget: count the complete request, leave room for the model’s response and any reasoning tokens, trim or retrieve context before it overflows, and persist important task state outside the rolling conversation. A context window is not a quota for how much history your application may store; it limits what can be used in a particular model request.
What a context window includes
A context window is the maximum number of tokens available to a single request. What counts toward it depends on the provider, model, and API. It can include instructions, conversation messages, tool definitions and results, retrieved documents, images or other inputs, and generated output. OpenAI says its context window includes input and output and, for some reasoning models, reasoning tokens; Anthropic describes the system prompt, messages, tools, and generated output as part of the context; Gemini uses a combined input-and-output limit. See the provider documentation for OpenAI conversation state, Anthropic context windows, and Gemini token counting and limits.
As an Amazon Associate I earn from qualifying purchases.
Tokens are not words. Token counts vary with the model and encoding, language, and content type. A transcript that looks short in a log or user interface can expand when formatted as a request, and multimodal inputs or tool schemas may contribute tokens that a text-only estimate misses. Do not assume a single context-capacity number applies across models or API surfaces.
How to measure the request the model will receive
- Build the complete next request. Include system and developer instructions, the messages you plan to send, tool definitions, schemas, tool results, retrieved content, and any images or files. Count the request in the format used by the actual API call, rather than counting only the visible conversation text.
- Use the matching provider and model’s counting method. OpenAI’s Responses input-token counting API includes structural tokens for message roles and boundaries; plain-text estimates can omit tools, schemas, images, and files. Gemini provides
count_tokensand model-information interfaces. Use the corresponding model and request shape, as described in the OpenAI token guide and Gemini token documentation. - Log actual usage after each call. Record the usage fields returned by the API, including input and output tokens and cached-token usage when provided. Compare those values with your estimates and investigate recurring gaps; the API’s accounting is more useful for calibration than word count.
Counting endpoints and model specifications can change. Check the current documentation for the model and API version you deploy instead of hard-coding a capacity based on a different model or snapshot.
How to budget for output and prevent overflow
Do not aim to fill the entire context window with input. The response also needs capacity, and some models use reasoning tokens within the context limit. Set the endpoint’s output limit deliberately and leave enough room for the expected answer and reasoning allocation. Output limits and total context limits are distinct: a request can fit while its configured output allowance is still too small for a complete response.
Set an application-level action threshold below the hard limit. There is no universal safe percentage: choose a trigger based on measured request sizes, expected response length, variability in tools and retrieved material, and how much interruption your workflow can tolerate. When the threshold is reached, reduce or compact context before sending another request. Check the model reference for current context and output limits; OpenAI’s reasoning-model guidance explains why reasoning-token accounting matters for applicable models.
What to do as conversation history grows
Remove material that no longer helps
Delete duplicated instructions, stale tool output, and earlier content that does not support the current task. Keep instructions and facts that still affect decisions. The aim is not simply to make the prompt shorter; it is to preserve the information needed for the next action.
Retrieve or select relevant information
For large knowledge bases or long documents, select or retrieve the material relevant to the current question instead of repeatedly placing everything in every request. If the same large input must be reused, evaluate provider caching where available. Google’s long-context guidance discusses caching for repeated context and cautions that retrieval performance and cost depend on the workload. A larger window can reduce how often material must be dropped, but it does not eliminate request cost, latency, or the need to surface the right information.
Split oversized inputs and summarize selectively
Divide inputs that are too large for one useful request into manageable parts, then carry forward the findings that matter. When older conversation must remain coherent, summarize it while preserving concrete facts, decisions, constraints, and unresolved questions. A summary saves space only if it retains what later steps need; check critical details against their source rather than trusting a lossy summary.
When and how to use provider compaction
Compaction replaces or summarizes accumulated conversation state so later requests can continue with less context. It is provider-specific, and its returned state may not be ordinary readable text. Follow the provider’s continuation rules exactly; do not mix one provider’s compaction behavior with another’s history-management flow.
| Provider or feature | What it does | Continuation detail |
|---|---|---|
| OpenAI Responses API | Can compact server-side after a configured rendered-token threshold or through a separate compact operation. See OpenAI compaction. | The returned compaction item is opaque. With input-array chaining, append returned items and you may drop items before the latest compaction item. With previous_response_id, pass only the new user message; do not manually prune the history for that flow. See OpenAI conversation state. |
| Anthropic threshold compaction | The documentation describes a beta threshold strategy configured through context_management.edits. See Anthropic compaction at a token threshold. |
Continue with the returned compaction block on later requests and drop prior blocks as directed. Confirm current beta and model coverage before relying on it. |
| Gemini | The cited token and long-context documentation covers counting, model information, and caching for repeated context; it does not establish an equivalent compaction workflow. See Gemini token documentation and Gemini long-context guidance. | Use the documented counting and model-information interfaces; check current Gemini documentation for the context-management options supported by the model and API you use. |
Do not assume compaction is lossless or portable. Validate that essential facts and instructions survive it, and verify feature availability and semantics against current provider documentation before adopting a workflow.
Recommended Free Tools
How to preserve agent state across calls and sessions
Conversation history is useful working context, but it is a poor sole source of durable task state. Save important state in an application session, database, or explicit artifact so a process restart, compaction, or new conversation does not depend on the model recalling every earlier message. A useful state record captures:
- The objective and current task status.
- Constraints, preferences, and decisions already made.
- Source-of-truth references or records needed to verify facts.
- Completed work, open questions, and the next concrete actions.
Keep the artifact concise and structured enough to reload into a fresh session. Update it when consequential decisions or task progress change, and have the application check for required fields before continuing. OpenAI’s Agents SDK session documentation describes persistent session history and OpenAIResponsesCompactionSession, which can replace longer stored history with a shorter item list. Its documented default trigger is item-count-based and can be customized with token counts or other heuristics. Avoid combining that compaction session with a server-managed conversation session that follows a different history flow.
How to choose a context-management strategy
Compare strategies against the requirements of your workflow, not just the advertised window size:
- Request accounting: Does the method count the complete API request, including non-text inputs and tools?
- State retention: Does it preserve the facts and decisions required for the next step?
- Headroom: Can it leave room for the expected output and applicable reasoning tokens?
- Cost and latency: What happens when large context is sent repeatedly, and can selection or caching reduce repeated work?
- Portability: Does the approach depend on provider-specific counting or opaque compaction items?
- Recovery: Can the agent resume after a process or session interruption using an explicit state record?
- Implementation effort: Can the application validate state and handle partial or incomplete responses reliably?
What to monitor in production
Track actual input, output, and cached-token usage where returned, alongside incomplete outputs, latency, and cost. Alert before requests approach the hard limit, then inspect whether the chosen response—trimming, retrieval, summarization, or provider compaction—preserves the details required for the next action. Add application-level checks for required state before allowing an agent to continue. Treat incomplete generation as a recoverable condition: detect it, save current durable state, and retry or resume with a smaller request rather than assuming the model completed the task.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
