Debug a bad agent result by tracing one run from its root through model calls, tools, retrieval, memory, handoffs, and the final response. Find the first point where the observed state diverges from the intended task; then separate what the trace shows from what you can only infer about the model’s decision. A trace can reveal the sequence of events, but it does not automatically expose every memory store or prove why a model behaved as it did.
What counts as a useful agent trace?
A useful trace presents a run as an execution tree, not a pile of unrelated log lines. The root represents the run or session; nested activity shows which agent performed each model call, tool call, handoff, or subagent task. That context matters: a tool invocation is easier to interpret when you can see what prompted it and what happened after it.
As an Amazon Associate I earn from qualifying purchases.
The OpenAI Agents SDK documentation describes built-in tracing for LLM generations, tool calls, handoffs, guardrails, and custom events. OpenAI’s session tracing documentation describes model responses and tool calls as spans associated with the agent that performed them. Those documented capabilities are not a guarantee that every application-specific event—especially reads and writes in an external memory system—will appear automatically.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsKeep three categories separate as you investigate:
- Observed: recorded events and their order, such as a tool call and its returned result.
- Inferred: a possible explanation, such as the model choosing a tool because a prompt was ambiguous.
- Unobserved: state or behavior not captured by the instrumentation, such as a memory lookup that produced no event.
This distinction prevents a plausible explanation from being mistaken for something the trace actually proves.
How should you define and inspect a failed run?
Capture enough context to reproduce it
Start with a stable run or session identifier and record the application version, prompt or configuration version, model identifier when available, timestamp, input, and outcome label. Record relevant dependency versions too, including the index or retrieval configuration if the task uses RAG. This is an implementation recommendation, not a universal schema promised by the tracing documentation.
Label the outcome in concrete terms: for example, “returned the wrong account status,” “timed out before answering,” or “answered without the required evidence.” If possible, replay the same input against the same dependency versions. A replay with a changed prompt, model, corpus, or tool implementation may help test a hypothesis, but it is not a like-for-like reproduction.
Walk from the root to the first divergence
- Open the root run and confirm that its input and outcome match the incident.
- Follow the child events in order: model activity, tools, handoffs, subagents, and application events that were instrumented.
- At each step, compare the observed state or output with what the task required at that point.
- Mark the earliest divergence, then inspect the preceding event and the downstream use of its result.
OpenAI’s session observability guidance describes inspecting turns, tools, subagents, and traces. The precise fields available in a trace depend on the SDK, application instrumentation, and trace configuration.
Free tools Windows power users keep installed
One-click scans. No signup required.
How do you distinguish tool-selection errors from tool failures?
Inspect each invocation in the context of the agent or subagent that made it. A failed tool-related outcome can arise at several different points, and treating all of them as “the tool failed” sends debugging in the wrong direction.
| What to check | Question it answers |
|---|---|
| Tool choice | Was this the appropriate tool for the task, or should the agent have answered, asked for clarification, or chosen another tool? |
| Arguments and validation | Were the arguments complete, correctly typed, in scope, and accepted by validation? |
| Execution status | Did the tool run, fail, or time out? Were retries attempted, and what were their outcomes? |
| Returned result | What did the tool actually return, including any error or partial result? |
| Downstream use | Did the agent use the result accurately, ignore it, or misinterpret it? |
These checks separate a poor decision to call a tool from an invocation problem and from a correct tool response that the model handled badly. A trace can help establish the recorded call and response; the exact fields available depend on the SDK and its configuration.
How do you find where a RAG answer went wrong?
Follow the path from the corpus to the response. A bad answer may result from retrieving the wrong evidence, failing to retrieve relevant evidence, or generating an answer that does not faithfully use evidence that was retrieved.
- Confirm the source being searched. Check the intended corpus or index and its version. A correct query against the wrong or stale source can still produce a wrong answer.
- Inspect query construction and filters. Review the query sent to retrieval and any filters or scope restrictions that could exclude relevant material.
- Review retrieved passages and metadata. Check the actual chunks, their ranking, and available source or version information. Do not assume a passage was available to the model just because it exists somewhere in the corpus.
- Compare evidence with the answer. If relevant material was retrieved, examine whether the response used it accurately, cited it when required, or contradicted it.
- Compare with a known-good case. Use the same evaluation criteria to contrast the failing run with a successful example.
If relevant evidence is absent, investigate ingestion, chunking, query construction, filters, and retrieval or ranking. If it is present but the answer misuses it, focus on the generation step and how the retrieved context was passed to it. LangChain describes LangSmith visibility into RAG pipelines; that product overview does not establish a universal debugging standard or guarantee that a particular deployment exposes every field above.
How do you diagnose memory problems that do not appear in the trace?
Treat memory as application state that needs explicit instrumentation. Do not assume that a model-call trace records what an external memory system returned, why an item was selected, or whether a write succeeded.
Add an event or span around each relevant memory read and write. Where appropriate and safe, record:
- the operation type and outcome;
- a memory item identifier or safe hash rather than the full sensitive payload;
- the item’s source or lineage, version, and timestamp;
- the run that read or wrote it;
- why the item was selected and which scope or user it applied to.
Then check for the concrete failure modes: a needed item was missing, an item was stale, two items conflicted, or data from the wrong scope was selected. To investigate a bad answer, connect the memory event to the run that consumed the item rather than relying on a broad snapshot of current memory.
The reviewed OpenAI Agents SDK documentation supports custom trace events, and its tracing privacy documentation describes controls for captured data. Neither establishes automatic lineage for arbitrary memory stores. Memory instrumentation described here is an application-level recommendation.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →How can you verify a fix and prevent a repeat?
Turn each incident into a regression case with the original input, the expected tool or retrieval behavior, and a measurable success criterion. Preserve the relevant versions and compare traces when changing code, prompts, models, tools, or indexes; otherwise, a changed outcome may have several possible causes.
Track outcomes that help the team spot regressions, such as failure rate, latency, cost, and user feedback where available. LangChain’s LangSmith overview describes dashboard metrics including token usage, latency percentiles, error rates, cost breakdowns, and feedback scores. These are vendor-described capabilities, not independent evidence of product performance.
What should you compare when choosing tracing or evaluation tools?
Compare tools against your actual debugging workflow, not the length of a feature list. A product can offer useful run traces yet leave memory lineage to your application, or expose a RAG pipeline without guaranteeing the fields your incident review needs.
| Evaluation axis | What to verify | What the cited documentation establishes |
|---|---|---|
| Trace coverage | Can you capture model calls, tools, handoffs, guardrails, and application-defined events? | OpenAI Agents SDK documentation lists LLM generations, tool calls, handoffs, guardrails, and custom events as built-in trace events. |
| Hierarchy and context | Can you see which agent or subagent performed each model or tool step? | OpenAI session tracing documentation describes agent spans with nested activity. |
| RAG visibility | Can you inspect retrieval alongside generation, and does your deployment expose the fields your team needs? | LangChain describes visibility into RAG pipelines in LangSmith; field-level support should be verified for the specific deployment. |
| Interoperability | Can trace data be exported or connected to your existing observability systems? | OpenAI documents OTLP JSON export for session traces, with enablement and permission requirements; LangChain describes OpenTelemetry support. |
| Metrics and evaluation | Can you compare cost, latency, errors, and feedback across runs? | LangChain’s overview lists dashboard metrics including token usage, latency percentiles, error rates, cost, and feedback. |
| Privacy and access | What data is captured, how can payload capture be limited, and what permissions does export require? | OpenAI documents sensitive-data capture controls and trace-export permission requirements. |
How should you handle sensitive trace data?
Trace inputs, outputs, tool arguments, and custom events can expose sensitive information. Decide deliberately what to capture, who can access it, how long to retain it, and whether identifiers or references can replace raw payloads.
The OpenAI Agents SDK tracing privacy documentation says sensitive-data capture is enabled by default and describes disabling it so request input and response output are omitted from model spans. Check the current SDK settings and export permissions for your deployment before relying on a particular configuration; privacy settings can change what evidence is available during debugging.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
