An AI agent can return a convincing answer and still fail its task: it may have chosen the wrong tool, sent incorrect arguments, relied on stale information, or skipped a required human handoff. Production reliability is therefore a workflow problem, not just a response-quality problem. Diagnose the full run—from context and decisions through tool calls and runtime outcomes—and keep an approved artifact, not an agent’s remembered summary, as the record of what is known.
Why a good-looking answer can hide a failed workflow
In an ordinary text-generation check, a reviewer may see a plausible response and mark it correct. That does not establish that the agent reached it safely or efficiently. An agent might consult an outdated source, take an inappropriate action, or arrive at the right result through an unacceptable path. Google Cloud describes this kind of discrepancy as a “silent failure”: a correct output produced through an inefficient or incorrect process. Google Cloud’s discussion of agent evaluation explains why the process matters alongside the final answer.
As an Amazon Associate I earn from qualifying purchases.
For each workflow, define success in observable terms. Specify the expected outcome, which tools and actions are acceptable, what arguments must be correct, and when the agent must stop or hand the work to a person. “Be helpful” is not a testable completion criterion. A useful evaluation question from OpenAI is: “Did the agent pick the right tool?” OpenAI’s agent-evaluation guidance treats tool choice and the end-to-end workflow as things to assess, not details to infer from a polished answer.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsInspect the full trace to find where the run went wrong
A trace is an end-to-end record of a workflow’s model calls, tool calls, guardrail outcomes, and handoffs. Reviewing it lets a team locate the divergence: perhaps the model chose an unsuitable tool, the tool received the wrong parameters, a guardrail changed the path, or a handoff never happened. OpenAI’s guide describes using graders to score traces against structured criteria, so evaluation can examine the sequence of events rather than only the final text. See the agent workflow evaluation guide.
#1 Best Overall
- Model decisions: Did the agent select an appropriate next action for the task and the evidence available at that point?
- Tool calls: Was the tool choice appropriate, and did the call contain the required, correctly formed arguments?
- Guardrails and handoffs: Did controls fire as expected, and was a person brought in when the workflow required it?
- Completion: Did the workflow satisfy its defined success condition, rather than merely produce a plausible response?
Capture representative runs, including failures, so the trace can be compared with the intended path. Treat traces as operational data: decide who can access them, what sensitive information they may contain, and how long they should be retained. A trace supports diagnosis; it does not by itself make every action correct or turn a remembered summary into authoritative evidence.
Keep working context, reusable memory, and evidence separate
These three mechanisms solve different problems. Confusing them can make an agent appear to “remember” facts that were never verified, or lead a team to trust a compressed prompt as a durable record.
Rank #2
| Mechanism | What it is for | What it should not replace |
|---|---|---|
| Current-run context and compaction | Manage the information available during an active run when the context is finite; compaction can help the run continue. | A durable factual record or an approved artifact. |
| Reusable memory | Carry workflow lessons forward so later runs can reuse useful operational guidance. | Validated evidence for a particular fact, decision, or case. |
| Reviewed artifact | Preserve the human-reviewed result, its provenance, and any uncertainty as the source people rely on. | The agent’s mutable working context or an unverified memory summary. |
OpenAI’s example on memory and compaction distinguishes continuing a run from retaining lessons for later runs. The memory-and-compaction example is useful for understanding those separate roles. In an application, record where important claims came from and what remains uncertain in the reviewed output; do not present an agent’s recollection as proof.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Test the boundary your application controls
Deterministic tests are most useful for the behavior the application owns: orchestration, tool routing, handoffs, guardrails, retries, and session handling. The OpenAI Agents SDK provides in-memory testing support for those kinds of orchestration checks. When behavior depends on an external model, protocol, provider, or audio service, an in-memory test cannot establish how that dependency behaves in its real environment. Test that boundary with the relevant adapter or integration environment. The Agents SDK testing guide describes the SDK’s testing approach and its scope.
Rank #3
- Test the expected path. Provide representative inputs and assert the task outcome, tool choice, arguments, and any required handoff—not just the response text.
- Test failure handling. Exercise retries, tool errors, timeouts, guardrail outcomes, and incomplete outputs that the application is expected to handle.
- Test real dependencies where they run. Use integration coverage for external behavior that an in-memory orchestration test does not exercise.
- Keep observed failures as regression cases. When a production trace reveals a defect, turn the relevant scenario into a repeatable test where possible.
A passing orchestration test shows that the tested application path behaves as specified under the test’s conditions. It does not prove that an external service will always behave the same way, nor does a response-only assertion prove that the agent followed an acceptable trajectory.
Build evaluations around the work the agent actually does
An evaluation set should represent ordinary tasks, edge cases, and failures seen in operation. Its criteria need to match the workflow: outcome correctness, tool selection, argument precision, acceptable process, and trustworthiness when conditions are adverse. Use automated checks for criteria that can be stated precisely, and human review for judgments that need context. Compare automated graders with human judgments and calibrate them when they disagree; otherwise, a grader can consistently reward the wrong behavior. OpenAI’s evaluation best-practices guide discusses representative data, task-specific criteria, production examples, and human feedback.
Rank #4
- Outcome: Was the requested task completed to its stated acceptance criteria?
- Trajectory: Did the agent use an acceptable process, including an appropriate source and action sequence?
- Tool use: Was the right tool selected, and were its arguments correct?
- Trust: Did the workflow preserve uncertainty, respect controls, and request human review when needed?
Run evaluations continuously as prompts, tools, models, and application logic change, and incorporate suitable production examples. Evaluation targets or sample datasets shown in guidance are illustrative; they are not measured evidence of a general production failure rate or a guaranteed reliability improvement. The reviewed material establishes no such prevalence or improvement figure.
Separate runtime errors from reasoning and evaluation failures
A workflow can fail before a useful decision is made, during a tool call, or after the run exceeds its operational limits. Track these as distinct signals rather than grouping every unsuccessful run under “bad answer.” The Agents SDK documentation identifies issues including turn-limit exhaustion, model timeouts, malformed or incomplete outputs, unexpected tool failures, and tool timeouts. The SDK guide to running agents covers execution behavior and relevant failure cases.
- Turn limit reached: The run used up its allowed turns without meeting the completion condition. Check for looping or an unclear stop condition.
- Model timeout: A model call did not finish in the available time. Distinguish this from an invalid model output or a failed downstream tool.
- Malformed or incomplete output: The returned content did not satisfy the application’s expected format or completeness requirements. Validate output at the boundary where it is consumed.
- Tool failure or timeout: The tool errored or did not respond in time. Record which call failed and whether the workflow safely retried, stopped, or handed off.
For work that must survive process restarts or wait for a long human approval, assess whether durable orchestration is needed. A short request-response interaction has different recovery needs from a workflow that must resume later. Choose execution and recovery behavior to fit the workflow’s failure tolerance; the cited SDK material does not establish one universally best architecture.
A practical production diagnosis, from symptom to fix
- Write down observable success. Define the required result, acceptable tool behavior, and conditions for a human handoff before judging whether a run passed.
- Collect representative end-to-end traces. Include ordinary runs and failures, then inspect model calls, tool calls, guardrails, and handoffs to identify the first point where the actual path departed from the intended one.
- Reproduce application-owned behavior deterministically. Add tests for orchestration and failure handling; use a real integration environment for dependencies whose behavior is outside the application boundary.
- Expand the evaluation set with task-specific cases. Cover routine inputs, edge cases, and observed failures. Score outcome and trajectory, including tool choice and argument correctness, and calibrate automated graders against human review.
- Audit what the agent carries forward. Keep active-run context, reusable workflow lessons, and reviewed factual records distinct; retain provenance and uncertainty in the authoritative artifact.
- Monitor execution failure categories separately. Track turn exhaustion, timeouts, invalid or incomplete output, and tool errors. For long-running work, decide whether recovery across restarts and approval waits is a requirement.
OpenAI Evals platform timing is product-specific
OpenAI’s evaluation guidance says its Evals platform will become read-only for existing users on October 31, 2026, and is scheduled to shut down on November 30, 2026. These dates concern that specific platform, not agent evaluation as a practice or every evaluation tool. Teams relying on it should check the current OpenAI guidance for any change to the schedule. OpenAI’s evaluation best-practices page contains the platform notice.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
