To debug an AI agent, start with one reproducible run that failed, inspect its end-to-end trace, then follow the first wrong decision into the application code. Grade representative traces against explicit criteria and save recurring failures and expected behavior in a dataset you can rerun after changes. Before tracing real users, decide what prompts, outputs, tool data, and audio may be captured.
Start with a reproducible failing run
Choose an actual run that demonstrates the problem. Record the user request, the expected outcome, what happened instead, the relevant agent and tool versions, and the trace identifier. This gives you a concrete case to inspect and rerun; a broad prompt rewrite before locating the failure can change behavior without explaining it.
Read the trace as a sequence of decisions
An end-to-end trace should let you follow the workflow through model calls, tool calls and their results, handoffs, guardrail events, and custom spans around important application code. OpenAI documents this trace model for its Agents SDK, whose tracing is enabled by default in its normal server-side path. See Agents SDK tracing and OpenAI’s integrations and observability guide.
Read from the beginning and identify the first point where the run diverged from the expected path. That may be the model’s interpretation, its tool choice, the tool’s result, a missing or inappropriate handoff, a guardrail, or another application boundary. A final answer that looks wrong may have originated several steps earlier.
- Compare the model call’s input and output with the task and instructions.
- Check the selected tool, its arguments, and the result returned to the agent.
- Follow any handoff and check whether control went to the right agent or stage.
- Inspect guardrail events and relevant custom spans for the application’s behavior at key boundaries.
Follow the failing boundary into your code
A trace narrows down where to investigate; it does not, by itself, prove root cause. Follow the event into the code that built the prompt, chose or validated a tool, transformed its output, routed work, or accepted the final response. Check the actual inputs and outputs at that boundary rather than assuming the model alone caused the failure.
If the trace lacks context, add a custom span or ordinary structured logging around the important application step. OpenAI’s documentation describes custom spans, but instrumentation should be treated as evidence to inspect alongside the code and run—not as proof of causation.
Grade traces against explicit behavior
Once you can identify the relevant part of a run, define criteria tied to the task. For example: Was the correct tool chosen? Was a handoff appropriate? Did the workflow follow its instructions and safety constraints? Apply those criteria to representative traces rather than relying only on whether the final answer sounds plausible.
OpenAI describes grading selected traces and using the results to target prompts, tool surfaces, routing, or guardrails. Its guide defines trace grading as “the process of assigning structured scores or labels to an agent’s trace—the end-to-end log of decisions, tool calls, and reasoning steps—to assess correctness, quality, or adherence to expectations.” Read OpenAI’s trace-grading guide and its guide to evaluating agent workflows.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Turn recurring failures into a reusable dataset
Individual trace inspection is useful for understanding an early failure. To compare changes and catch regressions, collect representative successes, failures, and edge cases with an expected outcome or a grading rubric. Rerun the same evaluation after changing a prompt, model, tool, or routing rule. A repeatable set of cases makes it possible to compare versions rather than relying on memory of a few runs.
OpenAI presents datasets and eval runs as a way to benchmark changes and compare prompts over time. The workflow is simple: preserve a meaningful case, state what acceptable behavior looks like, then run the evaluation again when the workflow changes. The result can show whether a fix improved the cases it targeted and whether it affected other examples in the set.
Decide what trace data you can safely collect
Traces can contain sensitive information. OpenAI’s Agents SDK documentation says generation spans may store LLM inputs and outputs, function spans may store function inputs and outputs, and audio spans include encoded input and output by default. The documented trace_include_sensitive_data setting can disable certain text capture; audio has a separate setting. Check the active SDK version and configuration, as well as the exporter and backend, before enabling tracing with real user data. Review access, retention, and redaction requirements for your deployment.
When a hosted observability platform may help
A hosted tool is optional: the core debugging loop is to inspect a run, grade behavior, and repeat evaluations against a dataset. If you are comparing platforms, look at trace coverage, framework and OpenTelemetry support, evaluation methods, human review, deployment choices, and data residency. Product descriptions are vendor claims, not an independent comparison.
LangChain describes LangSmith observability as supporting a range of frameworks and OpenTelemetry, with dashboards for token usage, latency, errors, cost, and feedback. Its LangSmith evaluation page describes curated datasets, online evaluation, multiple grader styles, and human review, along with managed, BYOC, and self-hosted arrangements. Verify current capabilities and data-handling terms against your requirements.
An OpenAI cookbook example also demonstrates tracing and feedback with Langfuse, but that cookbook is archived and may not reflect current compatibility. Treat it as a starting point to investigate, not a current integration guarantee: Evaluating Agents with Langfuse.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
