Debug a failing AI application by turning the symptom into a reproducible case, tracing one run across every boundary, isolating the first divergence, and testing one targeted change against the original failure. Do not begin by swapping models or adding retries: first establish what happened, where it happened, and whether the operation already completed.
The workflow below applies to agent systems, retrieval pipelines, tool-using assistants, and ordinary model API calls. OpenAI dashboards, traces, and error fields are used as concrete examples; the diagnostic method is provider-neutral.
As an Amazon Associate I earn from qualifying purchases.
1. Capture a reproducible observation before changing code
Write down one complete example while the failure is still visible. Record the exact user input or event that triggered it, the expected result, the observed result, and the timestamp with timezone. Mark whether the behavior repeats, occurs intermittently, or cannot currently be reproduced. Preserve a request ID, session ID, job ID, or your own correlation ID if one exists.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Also record the execution context: model and version, service tier, project or tenant, endpoint, deployment and build identifier, prompt or policy version, tool and retrieval configuration, dependency versions, and relevant traffic conditions. List recent changes to prompts, model selection, tool schemas, indexes, credentials, network infrastructure, dependencies, and deployment settings. This inventory is a diagnostic aid, not proof that the latest change caused the incident.
Define expected and observed behavior precisely
- Expected: the contract the system should satisfy, such as a schema-valid JSON object, a completed tool action, or a response within a stated deadline.
- Observed: the exact output, empty result, timeout, exception, partial stream, duplicate action, or user-visible message.
- Reproduction: the smallest input and sequence that still produces the symptom, including whether a fresh session, a particular user, or a particular data record is required.
A statement such as “the agent failed” is not testable. “For project P, model M, endpoint E, requests containing tool action T returned an empty final message between 14:05 and 14:17 UTC, while the same prompt without T succeeded” is testable.
2. Narrow the scope with matched comparisons
Separate affected traffic from unaffected traffic before drawing conclusions. Compare like with like: the same project, model, service tier, endpoint, deployment, and time window. In OpenAI’s service-health troubleshooting guidance, filtering to the affected project and a single model and tier prevents unrelated traffic from hiding the pattern.
Use rates and distributions rather than raw totals. A busy project naturally has more errors than a quiet one, so calculate errors divided by requests. For latency, inspect percentiles such as P50, P75, and P95 instead of relying on an average that can conceal a long tail. Keep the filters identical when comparing provider health data with your own usage records.
| Comparison axis | Question it answers | Useful evidence |
|---|---|---|
| Affected versus unaffected project or tenant | Is the problem isolated to configuration, data, or permissions? | Error rate, latency percentiles, deployment and credential differences |
| One model or tier versus another | Does the failure follow a model, capacity class, or routing choice? | Status codes, completion status, time to first token, usage records |
| Time windows | Did the symptom begin with an incident, release, or traffic change? | Minute-by-minute request counts, error percentages, deploy markers |
| Endpoint or region | Is one network or service path affected? | Provider response, proxy logs, load-balancer metrics |
| Build or prompt version | Did application behavior change without a provider error? | Trace outcomes, validation failures, representative examples |
If your client reports errors but the provider’s health or usage records show no corresponding requests, investigate the local path first: DNS, proxy, TLS, firewall, load balancer, client timeout, queue, or a dropped task may have prevented the request from arriving.
3. Follow one run from intake to final output
Choose a representative failed run and inspect it in sequence. A useful trace has a single correlation ID and records each boundary:
- Request intake: timestamp, authenticated project or tenant, endpoint, model, timeout, and a safe summary of the input.
- Model generation: request and response status, duration, token or usage data where available, finish reason, and validation result.
- Tool execution: tool name, arguments after redaction, start and end times, result status, timeout, and returned shape.
- Handoff or orchestration: the agent or worker that ran next, the state it received, and the transition decision.
- Guardrails and validation: policy checks, schema parsing, required-field checks, and any rejection or repair attempt.
- Final delivery: the output sent to the user, queue, database, or downstream API, plus whether the application discarded or replaced it.
OpenAI tracing documentation describes records containing inputs, outputs, duration, and status. Agent SDK traces can additionally include model generations, tool calls, handoffs, guardrails, and custom events. In a dashboard, search and filter by session, model, status, or date to find the run and then inspect the first incomplete or contradictory span.
What to look for in the trace
- A request that ends before a model response or a tool result.
- A successful model response followed by a parser, schema, or state-transition failure.
- A tool that returns an empty, stale, or differently shaped value.
- A handoff that names an unavailable worker or omits required context.
- A retrieval step that returns no documents even though the user’s record should match.
- A valid final response that fails the task’s correctness criteria.
- A response that exists in the trace but is never delivered because application code drops it.
If you do not have trace-level telemetry
Add structured events at every boundary rather than scattering free-form print statements. Include the correlation ID, event name, monotonic start and end times, status, safe input and output summaries, tool name, retry count, and validation result. Redact secrets, access tokens, and sensitive customer content; follow your organization’s retention and privacy rules.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsVerify what telemetry is actually available. OpenAI’s Agents SDK documentation notes that tracing can be disabled and is unavailable for organizations using its Zero Data Retention policy. In those environments, retain an equivalent, privacy-reviewed event trail in your own system or use the provider’s permitted metadata without assuming a missing trace means a missing request.
4. Classify the first failure layer
Do not treat every bad answer as a model problem. Identify the earliest boundary at which the run diverges from the expected path.
| Layer | Typical symptom | Evidence to inspect | Likely next action |
|---|---|---|---|
| Application path | Task disappears, state changes incorrectly, or a valid response is discarded | Worker logs, queue acknowledgements, state transitions, exception stack | Fix control flow, persistence, or exception handling |
| Model or prompt behavior | Request succeeds but output misses criteria or violates a schema | Prompt and model version, trace output, grader results, affected examples | Change the smallest relevant prompt, model, or constraint and evaluate |
| Tool or dependency | Timeout, stale data, empty retrieval, changed response shape, or downstream error | Tool request and result, dependency status, latency, schema version | Repair integration, timeout, data, or dependency contract |
| Request or configuration | Rejected request, authentication failure, quota or billing error | HTTP status and structured fields such as error.code, error.message, and, when present, error.param |
Correct input, credentials, limits, or account configuration |
| Network or provider path | Client timeout, connection failure, or provider incident correlation | Client, proxy, load-balancer, and provider records; request presence at the provider | Repair local networking or wait for and monitor provider recovery |
This classification is a working framework, not an exhaustive taxonomy. A single user-visible failure can contain more than one defect; fix the earliest causal defect first, then check whether later symptoms disappear.
5. Turn the evidence into a falsifiable hypothesis
Use one sentence with a trigger, mechanism, and symptom: When [trigger] occurs, [component] produces [observable mechanism], which leads to [user-visible symptom]. For example: when retrieval returns zero documents, the orchestration path treats an empty list as a successful context and sends a prompt with no evidence, which produces a confident but unsupported answer.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchTest the hypothesis against evidence that could disprove it. Compare affected and unaffected runs, versions, traffic windows, dependency responses, and trace completeness. If the suspected tool is healthy in successful and failed runs, it is less likely to be the first cause. If failures begin only after a parser version change and the model output is unchanged, the parser deserves priority over prompt edits.
Change one relevant component at a time. Simultaneously changing the model, system prompt, tool schema, timeout, and retry policy may make the symptom disappear while leaving the mechanism unknown. Keep the original reproducer unchanged so every candidate fix can be compared fairly.
6. Recover safely instead of masking the defect with retries
Before resending a request, determine whether the original operation completed. A timeout can occur after a provider or tool has committed a side effect. Repeating it may create a duplicate message, purchase, ticket, database write, or workflow transition. Use idempotency keys or an outcome lookup where the operation supports them.
| Condition | Retry decision | Required control |
|---|---|---|
| Rate limit, overload, transient timeout, or temporary service failure | Retry only when the operation is safe to repeat and the outcome is unknown or confirmed incomplete | Honor Retry-After, cap attempts, and impose an overall deadline |
| Invalid input or schema error | Do not retry unchanged | Correct the request and add validation before submission |
| Invalid or expired credentials | Do not retry repeatedly | Repair authentication and verify the intended project or tenant |
| Billing, quota, or account restriction | Do not use retries as a remedy | Resolve the account or capacity constraint |
| Unknown side-effect outcome | Check status before replaying | Query the operation, use an idempotency key, or require manual reconciliation |
A bounded retry policy needs both an attempt limit and a time deadline. Record each attempt and the final disposition so a later trace distinguishes one failed operation from several duplicated operations. Alert on retry exhaustion rather than returning a silent empty answer.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →7. Apply a targeted fix and run a regression check
Preserve a representative input set before editing the system. Include the exact reproducer, nearby normal cases, edge cases around the suspected boundary, and at least one case that previously succeeded. Define measurable checks before the change: schema validity, required fields, tool success, task correctness, error percentage, latency percentiles, or time to first token.
- Run the baseline set and save outputs, traces, status codes, and latency measurements.
- Change one component tied to the hypothesis, such as a parser guard, timeout, tool contract, prompt constraint, or model setting.
- Re-run the exact triggering input and confirm that the hypothesized mechanism no longer appears.
- Run the complete representative set, including adjacent normal and failure cases.
- Compare the same metrics and inspect new failures or trade-offs.
- Deploy gradually when possible, keeping the old build identifiable for rollback.
For behavioral defects, inspect traces while debugging and use explicit graders for regressions. OpenAI’s agent-evaluation guidance recommends starting with traces when behavior is still being debugged, then using structured graders to expose failure modes. The OpenAI Cookbook describes an iterative analyze-measure-improve evaluation cycle: diagnose examples, quantify the baseline, make a targeted improvement, and measure again.
Behavioral and operational checks
- Behavioral: task completion, factual or policy criteria, schema validity, correct tool choice, and correct handoff.
- Operational: request error percentage, P50/P75/P95 latency, timeout rate, retry exhaustion, queue age, and provider-versus-client request counts.
- Safety: no duplicate side effects, no secret leakage in logs, and no bypass of validation or authorization.
A fix is not proven by one successful manual run. It is supported when the original failure is gone, the mechanism is no longer present in traces, the representative set meets its criteria, and operational impact remains acceptable.
8. Write an incident record that another engineer can verify
Report the cause and evidence, not just the patch. Include:
- Trigger: the input, event, release, traffic condition, or dependency change that activates the defect.
- Impact: affected users, projects, endpoints, time window, and user-visible behavior.
- Mechanism: the component and state transition that produced the symptom.
- Evidence: correlation IDs, trace spans, structured errors, logs, metrics, and provider or network records.
- Correction: the smallest change that addresses the mechanism, plus any configuration or data repair.
- Verification: the original reproducer result, regression-set measurements, deployment scope, and monitoring plan.
- Remaining risk: cases not covered, known trade-offs, and the rollback condition.
Keep links between evidence and claims explicit. A reviewer should be able to start with the incident statement, open the cited run, identify the first divergence, and reproduce the post-fix check without guessing which log line matters.
9. Use provider incident reports as context, not as your diagnosis
An official OpenAI status write-up about elevated errors across ChatGPT, Codex, and the API illustrates why symptoms can amplify. Internal checks opened new connections across multiple services; failed checks generated additional attempts. The reported remediation included fixing expensive database queries and expanding network capacity. That example demonstrates how a health-check or connection mechanism can create more load, but it does not establish the same cause in your application.
For your incident, correlate the provider timeline with your filtered project, model, tier, endpoint, and client metrics. A broad status message can explain a timing coincidence; only your request-level traces and network evidence can establish whether a particular run failed locally, at a dependency, or at the provider.
Quick Recap
10. A compact checklist for the next silent failure
- Save one exact input, expected result, observed result, timestamp, and correlation ID.
- Mark the affected project, model, tier, endpoint, deployment, build, and user scope.
- Compare error percentages and latency percentiles with matched unaffected traffic.
- Open one end-to-end trace and find the first incomplete or contradictory boundary.
- Inspect HTTP status and structured error fields before interpreting the model output.
- Correlate application, proxy, load-balancer, dependency, and provider records.
- Classify the layer and write a trigger-mechanism-symptom hypothesis.
- Check side-effect completion before any retry; honor
Retry-Afterand enforce limits. - Change one relevant component and rerun the original reproducer.
- Run the representative regression set, compare baseline metrics, and record the evidence.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
