The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Structure an agent’s context in layers: keep durable goals and rules in its instructions, pass the current task and relevant conversation state in the model-visible history, maintain selected long-term facts as memory, and retrieve changing or extensive knowledge only when needed. Keep application state separate until the model needs a specific part of it, and treat retrieved content as untrusted input. Context is not just a prompt: it is the information and capabilities assembled for each step of an agent’s work.
How do I structure context for an AI agent?
Start by deciding what the model needs for this particular call, rather than sending every available fact. An agent may use several sources of context, but they differ in purpose, update frequency, and risk. The application can hold data the model never sees; it becomes model-visible only when the application supplies it through the interaction.
As an Amazon Associate I earn from qualifying purchases.
OpenAI’s Agents SDK documentation puts the distinction plainly: “When an LLM is called, the only data it can see is from the conversation history.” In that SDK framing, instructions, run input, tools, and retrieved information can all contribute through the conversation history. Anthropic describes the broader practice as context engineering: curating the full set of tokens available at inference, then refining it as a multi-step agent accumulates new information.
| Context source | Best role | Design consideration |
|---|---|---|
| Instructions | Stable goals, behavioral policy, constraints, and output requirements | Keep them durable; avoid embedding transient facts or whole reference documents. |
| Application and runtime state | Dependencies, authorization, identifiers, and current structured state | Not automatically model-visible; expose only the portions needed for the task. |
| Conversation input and history | The immediate task and relevant recent turns | Long histories may need pruning or summarizing to preserve useful context. |
| Persistent memory | Selected user preferences, durable learnings, and compact notes | Maintain it, check freshness, and resolve conflicting updates. |
| Retrieval and tools | Large, changing, or on-demand external knowledge and actions | Check relevance and provenance; treat returned content as untrusted. |
What should go in an agent’s memory versus its prompt?
Use instructions for how the agent should behave across tasks, not as a storage bin for everything it might need. Put the user’s immediate request and the conversation details that affect the current answer in the current model-visible history. Use persistent memory for a small set of durable facts worth carrying across runs, and retrieve larger or frequently changing knowledge when a task calls for it.
Instructions: durable behavior
Instructions are the right place for stable objectives, boundaries, required workflows, and response-format expectations. They should tell the agent how to handle a class of tasks. A temporary project status, a long policy manual, or a user’s latest request usually belongs elsewhere: these may change, be needed only sometimes, or be too large to include on every call.
Conversation history: the task in progress
Include the current request and only the earlier turns needed to interpret it. In a multi-step run, preserve decisions, unresolved questions, and evidence that still affects the next action. If the conversation grows, summarize or prune deliberately; an unfiltered transcript can consume context without making the next step more informed.
Persistent memory: selected facts that survive a run
Memory is useful when a fact is likely to matter again, such as a stable preference or a compact project note. It is not a substitute for the current source of truth. Old notes can become stale, and later information can conflict with them; prefer current verified state when that happens.
There is no single standard memory format. OpenAI Agents SDK documentation describes extracting summaries and raw notes from a conversation, then consolidating them into a more usable layout. AWS Prescriptive Guidance recommends combining structured state and recent dialogue with summaries and long-term-memory retrieval. These are implementation patterns, not a requirement to adopt one schema. Whatever format you choose, make notes easy to update and distinguish durable facts from uncertain or time-sensitive ones.
How should context be assembled for each agent run?
Build the context deliberately at each model call. The right package can change between steps: a tool result may answer one question, while the next step needs a different source or a narrower summary. A practical run sequence is:
- Read the current task. Identify the outcome requested, constraints, and any immediate ambiguity.
- Load the durable instructions. Apply the stable goals and behavioral policy that govern this agent.
- Expose only necessary application state. Select relevant identifiers, permissions, or structured values in the application; do not assume the model can inspect state held elsewhere.
- Include relevant conversation state. Pass the current request and the prior turns, decisions, or concise summary needed to continue correctly.
- Retrieve missing external information when needed. Use a tool or retrieval step for large or changing sources, then select useful evidence rather than forwarding everything returned.
- Check the assembled context and action risk. Verify relevance, source, freshness, and whether untrusted content could influence a consequential tool call.
- Evaluate the result and update state selectively. Assess retrieval quality separately from the model’s response, then preserve only useful durable memory.
This is a repeated assembly process, not a one-time prompt-writing exercise. The agent’s state and information needs can change at every turn.
When should an agent use retrieval instead of including knowledge directly?
Use retrieval when relevant knowledge is too large, changes often, or is needed only for some tasks. Retrieval systems commonly split a corpus into chunks, find likely matches, and add selected passages to the model-visible context. The retrieved passages should support a specific task; dumping a broad search result set into the prompt can add noise as well as tokens.
Free tools Windows power users keep installed
One-click scans. No signup required.
Match retrieval to the kind of question
Semantic search using embeddings can find passages that are conceptually related even when the wording differs. Lexical matching, including BM25, can be useful for exact phrases, product codes, file names, or other identifiers that semantic matching may miss. Anthropic’s 2024 description of contextual retrieval discusses combining lexical and semantic retrieval, deduplicating matches, and reranking them as one approach—not as a universal recipe. Test the combination against the actual corpus and queries.
Anthropic reported that its Contextual Retrieval method reduced failed retrievals by 49%, and by 67% when paired with reranking. Those are results reported by Anthropic for its method in its 2024 publication; they are not a general expected improvement for every retrieval system.
Choose direct inclusion or retrieval by workload
For some knowledge bases below 200,000 tokens, Anthropic’s 2024 article says direct inclusion may be simplest in the Claude context discussed there. That is a model- and publication-specific example, not a universal threshold. Google’s Gemini API guidance notes that long-context performance can vary when a task requires finding multiple information targets, and that longer inputs can increase latency and cost. Evaluate representative tasks rather than assuming that a larger context window removes the need to select information.
Compare approaches using corpus size and update frequency, exact-match needs, retrieval relevance, latency, token cost, data sensitivity, and the consequence of an incorrect answer or action. Caching may help with repeated static context where supported, but model support and pricing change; check the current product documentation before estimating savings.
How do you know whether retrieval or the model is failing?
Test retrieval and response generation as separate stages. OpenAI’s API accuracy guidance identifies two distinct failure modes: retrieval may supply missing, irrelevant, or excessive context, and the model may still misuse context that is relevant. If those are measured together, it can be hard to tell whether to improve search, refine context selection, or change the model-side behavior.
- Retrieval check: Did the system find the needed source, return the right passage, and include enough evidence without flooding the context?
- Response check: Given the evidence actually supplied, did the model answer accurately and stay within the stated constraints?
- End-to-end check: Did the agent complete the task correctly, including any tool actions or required review?
Use representative questions, including exact identifiers, ambiguous requests, changing facts, and cases where the answer is absent from the corpus. Record which sources were retrieved and what reached the model so that retrieval misses are distinguishable from reasoning or instruction-following errors.
How should context design handle prompt injection and risky tools?
Retrieved pages, files, and tool outputs can contain hostile instructions as well as useful facts. Treat them as untrusted data, not as a new authority that overrides the agent’s instructions. Filtering suspicious text can reduce exposure, but it is not a complete security boundary: model defenses do not catch every attack.
OpenAI’s API security guidance recommends layered precautions such as using trusted integrations, choosing file sources carefully, logging or reviewing tool calls, separating public research from access to sensitive data in some workflows, and checking tool arguments with schemas or regular expressions. OpenAI’s 2026 security article emphasizes constraining the consequences of manipulation rather than relying on perfect detection.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Give tools only the capabilities and data access needed for the task.
- Validate tool arguments and enforce authorization in application code, not only through model instructions.
- Require review or confirmation for consequential outbound actions where appropriate.
- Keep sensitive operations separated from untrusted browsing or retrieval when the workflow permits.
- Log relevant inputs and tool calls so suspicious behavior and failures can be investigated.
These controls reduce risk; they do not guarantee that every malicious instruction will be detected.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
