The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →To show whether an AI agent improved, inspect a trace of the failure, turn the task into explicit grading criteria, then run the same evaluation dataset before and after a documented workflow change. Report the scores alongside sample size, examples, and traces. MCP can provide the agent with tools and context, but it does not determine whether an answer is correct—and a higher score alone does not prove what caused the change.
Start with the failed run, not a guess about the prompt
A trace records what happened in one workflow execution. In OpenAI’s Agents SDK, a trace can contain model calls, tool calls, handoffs, guardrails, and custom events. Use it to reconstruct the path from input to answer, rather than judging only the final response. The Agents SDK tracing guide describes trace contents and controls; the Tracing guide covers inspection in the dashboard, including MCP tool-call details.
As an Amazon Associate I earn from qualifying purchases.
For the failure in question, ask: Did the agent pick the right tool? Did a handoff happen when it should have? Did the workflow violate an instruction or safety policy? Did it correctly interpret the tool result and finish the task? For an MCP call, inspect the server and tool selected and the recorded arguments. These details can distinguish a bad answer caused by tool selection from one caused by incorrect handling of returned data.
Free tools Windows power users keep installed
One-click scans. No signup required.
One trace explains one observed run; it does not establish how the workflow behaves generally. Collect representative cases before changing the system so the fix is not tailored only to one example.
#1 Best Overall
Define what a good answer means
Convert the failure into criteria a reviewer or grader can apply consistently. Choose each grader to match the kind of claim being checked. OpenAI’s Graders guide describes several options:
- Exact or string checks: Use for required phrases, values, or structural elements where a precise match is appropriate.
- Similarity measures: Use when closeness to a reference answer is meaningful, not as a substitute for checking correctness.
- Model-based rubric grading: Use for judgment-based dimensions, such as whether an explanation addresses the user’s actual question. State the rubric clearly.
- Python checks: Use when a criterion can be evaluated programmatically.
- Combined graders: Apply multiple checks when one score would hide important distinctions, such as factual correctness versus required formatting.
Keep component scores visible. A composite can make a result easier to summarize, but it can also conceal a regression in one dimension behind gains in another.
Build a dataset you can reuse
Include the original failure, plus a representative range of inputs and behaviors the workflow is expected to handle. Preserve the inputs and the expected outcomes, references, or rubrics needed by the graders. A single anecdote can help diagnose a bug; it is not a reliable estimate of broader performance.
OpenAI’s Evaluate agent workflows guide describes using traces, datasets, and evaluation runs to assess agent workflows. The Evals API reference documents the API surface for configuring evaluations and runs. Check the current documentation for implementation details, which can change.
Make one interpretable workflow change
Change one element at a time where practical. Possible targets include prompt instructions, the tools exposed to the agent, routing, or guardrails. Recording exactly what changed makes it easier to connect a score shift to the workflow version being evaluated. If several elements change together, the comparison can still show that the overall workflow behaved differently, but it is harder to identify which change corresponds with a particular result.
Run the same evaluation before and after
- Run the baseline: Evaluate the existing workflow on the fixed dataset with the chosen criteria. Save the results and representative traces.
- Record the change: Note the workflow version and the specific prompt, tool, routing, or guardrail change.
- Run the revised workflow: Use the same dataset and graders. Keep other evaluation conditions consistent where possible.
- Compare by criterion: Show raw counts and scores for each dimension, along with sample size. Identify cases that improved and those that regressed.
If you report a percentage-point difference, show both underlying scores. For example, moving from 60% to 70% is a 10-percentage-point increase; it is not a 10% relative increase. There is no single percentage formula that fits every evaluation, so name the measure you use and provide the values behind it.
Explain the result without overstating causation
A useful report separates the measured finding from your explanation of it. Include the dataset size, criteria, baseline and revised scores, component-level changes, and representative examples or traces. Then describe the workflow change and explain how the examples are consistent with your interpretation.
For instance, a trace showing that the revised agent selected the intended MCP tool and used its returned data correctly is evidence about that run. If the same evaluation also shows improved results across representative cases, the combination supports a stronger account than a score alone. A before-and-after comparison still does not prove that one particular edit caused the improvement: other conditions may have contributed.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Choose the runtime and MCP connection for your deployment
OpenAI’s platform-specific options illustrate choices that matter beyond grading. The Agents guide compares the Agents API, Agents SDK, and Responses API. Consider where execution occurs, how much integration work you want to manage, how state is handled, and who executes tools. These are OpenAI-specific examples, not universal requirements for agent systems.
MCP standardizes how applications provide tools and context to LLM applications; it does not certify a tool’s trustworthiness or decide whether the agent’s answer is right. The Agents SDK MCP guide describes supported server approaches. The Integrations and observability guide discusses MCP wiring and observability. A hosted remote server and a server reached from the agent runtime can differ in reachability, network-boundary control, execution, and where approvals are managed. Choose based on your deployment’s connectivity and control requirements, and assess external MCP tools as interfaces that need appropriate oversight.
Check trace availability and sensitive data
OpenAI documents tracing as enabled by default in the Agents SDK’s normal path, with global, code-level, and per-run controls to disable it. The SDK documentation says tracing is unavailable for organizations using OpenAI APIs under a Zero Data Retention policy. Confirm that your organization’s policy and SDK configuration permit the traces your evaluation workflow depends on.
The tracing guide also documents a sensitive-data setting that can omit request inputs and response outputs from Responses model spans. Review what your traces contain and who can access them before using traces as report evidence. If sensitive fields are omitted, make sure the remaining evidence is sufficient to support the conclusions you publish.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
