Reliable production AI agents need bounded retries, safe recovery paths, saved progress, and visibility into the cost of the entire run—not just its first model call. Classify each failure before deciding whether to retry, switch to a fallback, or ask a person to intervene.
Choose a recovery action based on the failure
An agent run can fail at the model, a tool, a queue, an external service, or a later workflow stage. The same recovery action is not appropriate for every failure. Capture the HTTP status and structured error code or message, then identify whether the request needs correction, may succeed if repeated, or may already have produced an external effect.
As an Amazon Associate I earn from qualifying purchases.
| Failure type | Typical examples | Recovery action |
|---|---|---|
| Request or configuration problem | Invalid or oversized request, credentials or permissions problem, unavailable model or resource | Correct the request, access, or configuration before resubmitting. Repeating the unchanged request will not fix its cause. |
| Transient service problem | Throttling, temporary capacity or overload, timeout, temporary service error | Retry only if the operation is safe to repeat; honor Retry-After when supplied and stay within a deadline and attempt budget. |
| Persistent dependency or capability failure | A model, tool, or server remains unavailable, or cannot provide the required capability | Use a compatible fallback, degrade the feature, serve a suitable cached result, defer the work, or route it to human review. |
| Uncertain completion or side effects | A timed-out action may have completed, or a failed agent turn may include completed actions | Inspect saved state, events, and tool results before repeating. Use idempotency protection or an action ledger when the downstream operation supports it. |
For asynchronous or multi-turn systems, a failed turn does not necessarily mean that nothing happened. OpenAI’s agent guidance says to check whether actions have already completed before asking a failed turn to repeat work. Recovery code should also tolerate unknown error codes and missing optional fields rather than failing while trying to handle a failure.
Retry only safe, transient work—and set a hard limit
A retry is useful when a temporary condition may clear and repeating the operation will not create a duplicate action. Before enabling automatic retries, decide what the operation’s timeout means: a request can time out even though the service or downstream tool completed its work. For operations with external effects, check state or apply supported idempotency controls before attempting them again.
#1 Best Overall
- Honor server timing: when a response supplies
Retry-After, use it rather than immediately retrying. - Otherwise back off: use exponential backoff with random jitter, and cap each wait so retries fit within the operation’s latency budget.
- Set an attempt budget: distinguish total attempts from retries. AWS gives six total attempts—an initial request plus up to five retries—as an example, not a universal setting.
- Stop when the situation changes: OpenAI’s guidance says to stop automatic retries if the error changes or the retry limit is reached. Reclassify the new error instead of repeating the same recovery path.
Retry settings can count attempts differently. The cited AWS guidance notes that botocore’s total_max_attempts includes the initial attempt, while OpenAI and Anthropic SDK max_retries settings count retries. Check the selected SDK’s current semantics so a configured number does not silently allow more requests than intended.
Make the deadline fit the work
Set connection and read timeouts separately where the client allows it. A read timeout shorter than a valid long-running inference can trigger unnecessary duplicate work. The full deadline should cover the initial call, any permitted waits and retries, and the time needed to return or safely defer the result.
Prevent capacity errors from multiplying
Repeatedly retrying sustained 503 or 529 capacity errors can add load to an already strained dependency. Bound concurrency, rate-limit and queue requests, and defer or shed low-priority work when needed. Consider provisioned or cross-region capacity only when the service and deployment support it; those options do not replace limits on requests and retries.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Use fallback for persistent failures, not as an unbounded second retry loop
When a dependency remains unavailable or cannot perform the needed work, a fallback can preserve some useful service. Possible paths include a compatible alternate model or tool, a cached response that is still appropriate, a reduced feature set, a deferred job, or human review. The right choice depends on whether a degraded result is safe and useful for that task.
- Keep contracts compatible: alternate routes should preserve the output schema and downstream expectations. Validate a fallback’s result before handing it to the next stage.
- Make degradation explicit: do not present a partial or cached result as if it were a fresh, complete answer when that distinction matters.
- Stop switching when the budget is spent: set clear limits for retries and fallback attempts so an unavailable dependency cannot trigger a chain of repeated calls.
- Use circuit breakers: failure-rate and latency thresholds can prevent repeatedly calling a dependency that is already failing.
AWS’s Well-Architected Agentic AI Lens recommends testing fallback paths alongside primary implementations, including resilience tests that inject inference failures or inconsistent knowledge-base results. A fallback that has never been exercised may fail precisely when it is needed.
Persist progress and validate each workflow stage
Represent an agent task as stages with explicit inputs, outputs, and recovery boundaries. Save useful outputs when the workflow transitions between stages, then validate them before a later stage consumes them. If a late step fails, the system can resume from the last valid checkpoint rather than replaying the entire task.
- Record stage state: retain the inputs, outputs, status, and relevant tool results needed to determine where the task stopped.
- Validate before handoff: check that a stage produced the expected structure and required values before invoking a downstream tool or model.
- Resume at a safe boundary: continue from a validated saved result where possible; inspect external effects before repeating any action that may already have completed.
- Escalate judgment calls: route failures that cannot be safely retried or degraded to human review, with enough state and context for someone to decide what should happen next.
AWS’s Well-Architected Agentic AI Lens describes the maturity progression as moving from uniform retries without backoff or jitter toward staged, classified recovery with end-to-end tracing. Its guidance is implementation advice, not an independent comparison of service performance.
Limit agent loops, tokens, and concurrency
Retries are only one source of runaway work. An agent may make multiple model calls, delegate to subagents, carry prior outputs into later prompts, or invoke tools repeatedly. Set explicit recursion or turn limits, completion-token caps, and concurrency limits for each workflow.
Capacity planning must account for both request volume and token volume. AWS describes estimating requests per minute and tokens per minute from thread rate, invocation rate, input length, completion limit, and recursion limit. Those relationships help frame an estimate; they are not a universal benchmark. Its example assumes no prompt caching, and actual call counts and token lengths can vary because later calls may include earlier agent outputs and context.
Trace the full run and attribute its cost
Instrument the full path across model calls, tools, queues, delegated agents, and downstream services. A trace should let an operator see where the task spent time, which recovery route ran, and what work had already completed when an error occurred.
AWS CloudWatch documentation lists metrics including total and average invocations; total, average, input, and output token usage; average, P90, and P99 latency; errors and throttling; and cost attribution by application, user role, or user. Pair reliability and spend measures so an apparently successful workflow does not conceal expensive retries or wasted work.
- Reliability: success by stage, error class, latency distribution, retry and fallback rates, and tool availability.
- Usage and spend: tokens and cost per completed task or outcome, retries per task, and delegated or tool work.
- Waste: abandoned runs and no-op work where those measures apply.
OpenAI’s Agents API documentation notes that a task can require multiple model calls. Include input, cached input, output, and reasoning tokens; subagent calls; retries; and applicable tool, sandbox-compute, or third-party charges when calculating total task cost. Cached input remains billable, so a high cached-input percentage alone does not establish lower total cost. Recorded usage may be best-effort, null, or revised and is not necessarily a final bill.
Because provider features, quotas, SDK behavior, pricing, and availability change, verify the selected provider’s current settings and billing details before setting production limits. The appropriate retry, timeout, and cost budgets depend on the operation, user latency target, and business boundary for the workflow.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
