When a self-hosted agent fails, first determine what completed and what changed—not just whether the process is running. Preserve the run’s state, classify the fault, and retry only if it is likely transient and the work is safe to repeat. If bounded recovery fails, switch to a tested fallback or hand the incident to a person through a path that does not depend on the agent itself.
What should happen when an agent fails?
A reliable recovery path turns a failure into four things: recoverable state, a visible signal, a safe next action, and a human route when automation is exhausted. There is no universal alert threshold or 3 a.m. response deadline; set those according to the workload’s impact and the service objectives you have chosen.
As an Amazon Associate I earn from qualifying purchases.
- Detect: Alert on meaningful run or service failures, and on expected work that did not arrive or finish.
- Triage: Identify the affected request, turn, session, environment, workflow stage, or dependency. Use the error and its trace context to locate where the failure occurred.
- Protect state: Inspect saved outputs and completed side effects before replaying any work.
- Recover proportionally: Retry a known-transient fault within a finite budget. For persistent faults, stop retries and use a fallback, a human review queue, or an explicit stop with escalation.
- Hand off and learn: Give the responder enough context to continue safely, then use the incident or drill to improve the recovery path.
AWS’s Agentic AI Lens recommends decomposing agent workflows into stages, persisting outputs, and validating between stages. That design contains a failure to the affected stage instead of forcing the entire workflow to start again.
How do you avoid losing work or repeating side effects?
Before restarting, establish whether the run merely stopped or whether it stopped after doing some of its work. An agent may have saved a file, sent a message, updated a record, or called an external tool before its connection failed. Treat a timeout or disconnected run as an uncertain outcome until you check.
#1 Best Overall
- Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
- Find the run identifier and inspect its status, saved outputs, and last completed stage.
- Check the systems the agent acted on for changes already made.
- Decide whether the next action is safe to repeat. If it could create a duplicate or irreversible change, reconcile the existing result or require human review before replaying.
- For long workflows, save useful outputs at stage boundaries and validate them before moving on. Resume from the last verified stage rather than starting over when the design permits.
The exact status fields and recovery controls depend on the agent framework. For example, OpenAI’s Agents API documentation advises checking run status and saved work, and confirming whether an action completed before repeating it. Those API-specific instructions are not a universal contract for self-hosted agents.
Which failures should you retry?
Classify the fault before acting. A temporary network interruption or rate limit may clear; invalid configuration, a missing permission, or a malformed request generally will not improve through repetition. The error may occur at the agent, workflow, or dependency level, so follow trace context across tool calls and services rather than relying on a single process log.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Rank #2
- Easily store and access 5TB of content on the go with the Seagate portable drive, a USB external hard Drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
| Failure pattern | Safer response |
|---|---|
| Likely transient fault, such as a temporary service interruption | Retry with exponential backoff and jitter, within a finite attempt count or deadline. Honor a service’s retry guidance when available. |
| Persistent fault, such as invalid configuration or authorization | Stop retrying. Correct the underlying problem or escalate it to an operator. |
| Unclear outcome after timeout or disconnect | Inspect saved state and external side effects before deciding whether to retry. |
| Failure continues after the retry budget is spent, or the error changes | Stop automated retries and use the configured fallback or human handoff. |
Retries need both a limit and a cutoff. Exponential backoff spaces out attempts; jitter helps avoid many workers retrying at the same time. A deadline or attempt budget prevents a struggling dependency from consuming resources indefinitely or amplifying an outage. AWS’s agentic AI guidance discusses failure classification, retries, fallbacks, and retry anti-patterns.
For the OpenAI Agents API specifically, its documentation says to honor Retry-After where applicable, set a retry limit or deadline, and stop when the error changes or the limit is reached. Apply those details only when using that API; other systems may have different retry signals and semantics.
Rank #3
- Easily store and access 1TB to content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop. Reformatting may be required for Mac
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
What should happen after retries fail?
Decide the exhausted-retry behavior before an incident. The right choice depends on what the agent is doing: a cached or degraded response may be acceptable for low-risk work, while a consequential action may need a queue for human review. If there is no safe fallback, stop explicitly and escalate rather than implying success.
- Degraded response: Return a clearly limited result when it remains useful and safe.
- Human review: Preserve the run and send it to an operator with the context needed to approve, correct, or complete the work.
- Explicit stop: Mark the task as failed, alert the responsible person, and avoid claiming that unfinished work succeeded.
- Dependency cutoff: Stop sending work to a dependency that is failing. This limits the chance that one unhealthy component causes a wider failure.
Fallbacks must be exercised, not merely configured. Apache Airflow’s AI provider documentation describes implementation examples involving durable execution, retry policies, OpenTelemetry tracing, and provider fallback. These are options for that ecosystem, not requirements for every agent deployment.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsRank #4
- Easily store and access 4TB of content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
What should the on-call handoff contain?
A useful alert tells the responder what failed and where to start. Include an operational record that can be searched independently of the agent’s conversational output.
- Run or task identifier, start time, and current status.
- Failure stage, error code or message, and relevant trace or log context.
- Outputs saved and side effects completed so far.
- Retries already attempted, their outcomes, and whether the retry budget is exhausted.
- The next runbook step, escalation contact, and any safe manual action.
Correlate logs, metrics, and traces across asynchronous boundaries and tool calls. Metrics and alerts can tell you that something is wrong; trace context and an operational record help explain where the path broke and what the agent already did.
Best Value
- [Upgraded Version] - This external hard drive features a mirrored logo stripe combined with a striped anti-slip design, and the rounded corners of the casing make it easier to grip. The stripes also have a heat dissipation function, ensuring stable and fast data transfer.
- 【Ultra-thin and quiet】 - The motherboard adopts JMicron 578 noise-free solution, giving you a quiet working environment. Lightweight and portable size designed to fit in your pocket for easy portability.
- 【Ultra-Fast Data Transfers】 - Pairing this external hard drive with JMicron 578 solution USB 3.0 and USB 2.0 interfaces enables blazing-fast data transfer. It boasts theoretical read speeds of up to 125MB/s and write speeds of up to 103MB/s.
- 【Plug and Play】 - With no software to install, just plug it in and the drive is ready to use.The hard disk chip is wrapped with an aluminum anti-interference layer to increase heat dissipation and protect data.
- 【What You Get】 - 1 x Portable Hard Drive, 1 x USB 3.0 Cable, 1 x User Manual, Gift-type shell packaging ,Three-year manufacturer's warranty and free technical support services.
How do you make recovery work if the agent stack is down?
Keep the runbook and escalation route outside the infrastructure they are meant to help recover. If the agent host, orchestration layer, or monitoring path is unavailable, responders still need a way to find instructions, contact the owner, and restore service.
Free tools Windows power users keep installed
One-click scans. No signup required.
Define recovery objectives that fit the workload, document who owns each escalation, and rehearse the recovery path—including fallback and human takeover. AWS’s operational recovery guidance emphasizes tested runbooks, recovery objectives, central operational knowledge, and repeated exercises. Review meaningful incidents and drills, then update both the automated behavior and the runbook.
How to evaluate a recovery design
When choosing an implementation, compare the behavior it produces rather than assuming one framework or monitoring setup is universally best.
Quick Recap
| Question | What to verify |
|---|---|
| Does it preserve progress? | Useful outputs are persisted and validated at stages; retries do not blindly repeat completed side effects. |
| Does it distinguish errors? | Transient failures can be retried, while persistent or changed errors stop or escalate. |
| Are retries bounded? | There is a finite attempt count or deadline, a backoff policy, and a cutoff for an unhealthy dependency. |
| Is there a safe fallback? | The degraded response, human review path, or explicit stop is appropriate to the task’s impact. |
| Can responders follow the failure? | Traces and correlated logs cover tools and asynchronous boundaries, and the handoff includes completed actions. |
| Can people recover it independently? | The runbook, operational record, and escalation contacts remain reachable when the agent infrastructure is unavailable. |
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
