Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Make LLM calls survive 429 and overload responses by combining a provider-neutral error classifier, a bounded exponential-backoff loop, concurrency smoothing, and strict cancellation rules. Honor a valid Retry-After value, add limited jitter so workers do not retry together, and stop when an attempt or end-to-end deadline is reached. Retry only transient failures; billing, authentication, malformed-request, and other permanent errors need a fix rather than another request. This approach works across OpenAI, Claude, Gemini, Amazon Bedrock, and custom HTTP clients without assuming that every provider uses the same error code or SDK behavior.
What should you do when an LLM API returns 429 or overload?
Treat the response as a scheduling signal, not as permission to run an unbounded loop. First classify the status and provider error code, then wait for the provider’s requested delay or apply capped exponential backoff with jitter. Keep one retry budget for the entire operation, including retries performed inside an official SDK, and record enough context to diagnose whether you exceeded quota, ramped traffic too quickly, or hit temporary model capacity.
As an Amazon Associate I earn from qualifying purchases.
A robust call path has five layers:
- Classification: capture HTTP status, provider type/code, request ID, model, region, and whether a response has started.
- Admission control: queue work and smooth concurrency before requests reach the network.
- Retry policy: honor
Retry-After, otherwise use capped exponential backoff and jitter. - Budgeting: enforce maximum attempts and an absolute operation deadline.
- Observability: emit attempt, delay, outcome, and quota signals for every call.
Unsuccessful requests can count toward per-minute limits. An aggressive retry loop can therefore extend the throttle instead of clearing it.
Which failures are safe to retry?
Retry only errors that are plausibly temporary and for which repeating the request is acceptable. Status alone is not enough: provider-specific codes distinguish quota, acceleration, overload, and permanent configuration failures.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
| Condition | Typical meaning | Default action |
|---|---|---|
| 429 with a documented transient code | Rate, quota, or acceleration limit | Wait, reduce pressure, then retry within budget |
| 408 request timeout | The server or network did not complete in time | Retry if the operation is still within its deadline |
| Transient 5xx | Temporary server failure | Back off and retry a limited number of times |
| Documented overload code | Serving capacity is temporarily constrained | Back off; consider a supported alternate region or model |
| Billing, spend, credit, or usage-limit error | Account action is required | Do not retry; alert or route to remediation |
| 401/403 authentication or authorization error | Credentials or permissions are invalid | Fix configuration; do not retry blindly |
| Malformed parameters or unsupported model | Permanent request defect | Correct the request and send it once |
Keep the provider error type or code alongside the HTTP status. For example, a 429 caused by a hard account limit should not be treated like a short-lived acceleration limit, even though both use the same status.
How do OpenAI, Claude, Gemini, and Bedrock label these errors?
The providers expose similar signals but not identical taxonomies. Build one internal set of retry decisions and map each provider’s status and code into it.
Recommended Free Tools
| Provider | Rate or quota signal | Overload signal | Documented guidance |
|---|---|---|---|
| OpenAI | 429 with rate_limit_error or slow_down |
503 with service_unavailable_error or server_is_overloaded |
Follow Retry-After when present, lower traffic, and ramp gradually. A slow_down response can occur even when requests-per-minute and tokens-per-minute figures appear within limits. |
| Claude | 429 rate_limit_error |
529 overloaded_error |
Sharp usage increases can trigger acceleration limits. The official SDK retries transient failures twice by default with exponential backoff and honors retry-after. |
| Gemini | 429 RESOURCE_EXHAUSTED |
503 UNAVAILABLE |
Use exponential backoff with jitter, retry only transient errors, and cap the retry count. Its Python SDK documentation describes automatic retries for transient failures. |
| Amazon Bedrock | 429 ThrottlingException |
529 overloaded_error |
Use exponential backoff with random jitter, honor Retry-After, and use cross-Region inference where that feature is supported. |
Provider labels and SDK defaults can change. Treat the table as a mapping to verify against the current API contract you deploy against, not as a reason to match status codes with a regular expression.
What retry algorithm works with any HTTP client?
Use a single loop around the client call. The loop should parse a valid server delay first; only then should it calculate a local backoff. Add bounded jitter, cap the delay, and check cancellation before and during the sleep.
- Send one request with a request or idempotency key when the API supports it.
- Capture the complete failure signal: status, provider code, headers, request ID, model, region, and elapsed time.
- Classify the result. Continue only for 429, 408, a transient 5xx, or a documented overload condition.
- Read
Retry-After. Accept a valid delay or HTTP-date, and never sleep for less than the server-requested interval. - Choose a fallback delay when the header is absent or invalid:
min(cap, base × 2^attempt) + jitter. - Check the deadline. If the next sleep or request cannot finish before the operation deadline, return a timeout result instead.
- Sleep with cancellation support, then retry. Do not block a worker that has already been cancelled.
- Stop at the attempt limit and return a typed error containing the final provider response.
There is no universal “correct” base, cap, or attempt count. Choose them from your workload’s latency objective, token cost, provider guidance, and measured failure rate. A short interactive request may allow one or two retries; a batch job with a long deadline may allow more, provided its queue and budget can absorb the extra load.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Full, equal, and bounded jitter
Without jitter, workers that fail in the same millisecond wake together and create another burst. Full jitter selects a random value between zero and the calculated exponential delay. Equal jitter keeps half of that delay and randomizes the other half. Whichever variant you use, cap the final sleep and log the chosen value so you can tune it from real traffic.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Parsing Retry-After safely
Servers commonly express the header as a number of seconds or an HTTP date. Reject negative, nonsensical, or unreasonably distant values, then fall back to your capped schedule. Account for clock skew when an HTTP date is used, and still add only bounded jitter after waiting at least the requested interval.
retry_after = parse_retry_after(headers["retry-after"], now)
if retry_after is valid:
delay = retry_after + bounded_jitter()
else:
exponential = min(cap, base * (2 ** attempt))
delay = exponential + bounded_jitter()
if now + delay > deadline:
return deadline_exceeded
sleep_cancellable(delay)How do you prevent retries from making overload worse?
Backoff handles a failed request; admission control prevents a whole fleet from generating the next failure wave. Put a queue in front of the client, limit in-flight requests per provider and model, and release work at a controlled rate.
Smooth concurrency
- Use a bounded queue so producers cannot create infinite pending work.
- Apply a per-provider and, when needed, per-region semaphore.
- Use a token bucket or leaky bucket for request and token budgets.
- Prefer gradual ramp-up after an outage instead of restoring the old concurrency instantly.
- Spread scheduled jobs so they do not all start on the minute.
Account for token dimensions
Many services enforce more than requests per minute. Track input tokens, output tokens, daily limits, and any model- or region-specific quota that the provider exposes. A request can be under an RPM ceiling yet trigger a token or acceleration limit. Admission control should reserve estimated input and output capacity, then reconcile with actual usage after completion.
Use failover deliberately
Cross-region or alternate-model routing can help with a serving-capacity problem, but it does not erase an account-wide quota. Route only to a region or model whose capabilities, data residency, price, and quota you have already validated. Keep the same deadline and idempotency rules after failover.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
How should SDK retries and application retries be combined?
Assume an official SDK may already retry eligible failures. OpenAI’s official SDKs retry eligible 429 and 503 responses subject to settings; Claude’s SDK retries transient failures twice by default; Gemini SDKs retry transient errors by default. If your application adds another loop without accounting for those attempts, one logical call can generate many physical requests.
- Inspect the SDK’s default retry count, backoff, timeout, and retryable-code list.
- Choose one owner for the overall deadline: the application layer should normally enforce it.
- Either disable SDK retries and implement one policy, or include SDK attempts in your application’s attempt and elapsed-time budget.
- Expose per-attempt telemetry from the SDK if possible; otherwise count the call as an opaque sub-operation and use a conservative outer budget.
Nested retries also distort rate calculations. If three application attempts each trigger two SDK retries, the provider may receive up to nine requests for one user action. Set limits on total physical attempts, not just outer-loop iterations.
What changes when the request uses streaming?
Before a stream starts, normal HTTP retry handling applies. After a 200 response and SSE or other stream has begun, the request has crossed a different boundary: the provider may emit an error event after sending partial output. Automatically replaying it can duplicate work or side effects.
Before the first event
Classify connection failures, 429s, 408s, transient 5xx responses, and documented overload errors as you would for a non-streaming call. Retry only if no meaningful output has been consumed and the operation remains within its deadline.
After output has started
Persist the partial result and stream state. Resume only when the provider explicitly supports resumable streams and your protocol can identify the cursor. Otherwise surface a typed interruption to the caller, or replay only an explicitly idempotent operation whose duplicate output your application can safely discard.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Never treat a partially generated answer as proof that a side effect did not occur. For tool calls, payments, writes, or queue publications, require an idempotency key and an application-level deduplication record before allowing a replay.
How do idempotency and deadlines affect a retry?
A retry repeats the request, so safety depends on the operation, not merely the HTTP method. Pure text generation is usually easier to replay than a tool invocation that changes data. Attach a stable operation identifier to every attempt when the API supports idempotency, and make downstream tools reject duplicates.
- Deadline: carry one absolute deadline through queue wait, connection setup, backoff, response reading, and post-processing.
- Cancellation: propagate user cancellation and service shutdown into the sleep and HTTP request.
- Timeout type: distinguish connect, read, total-operation, and queue-wait timeouts in telemetry.
- Duplicate handling: store the operation ID and final outcome before acknowledging a job.
A retry that starts after the caller’s deadline is not resilience; it is hidden work that consumes quota and may return nowhere.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →What should you log and alert on?
Record one structured event per physical attempt and one final event per logical operation. At minimum include:
- logical operation ID and idempotency key;
- provider, model, region, and client or SDK version;
- HTTP status, provider error type/code, request ID, and whether streaming had started;
- attempt number, SDK-attempt count if available, selected delay, and total elapsed time;
- estimated and actual input/output tokens;
- queue wait, connection, server, and response-read durations;
- final outcome: success, deadline, cancellation, permanent error, or retry exhaustion.
Alert on rising retry rate, queue age, deadline-exceeded rate, and token burn—not only on raw 429 counts. A low 429 rate with rapidly growing queue latency can still indicate that admission control is failing. Sample response bodies carefully and redact prompts, personal data, and credentials.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
How do you troubleshoot a retry storm?
Start with the timeline rather than increasing the retry count. Identify whether failures began after a deployment, a traffic ramp, a model change, or a regional event.
- Check classification. Verify that permanent 4xx errors are not entering the retry path.
- Check physical attempts. Compare application attempts with SDK attempts to reveal multiplicative loops.
- Check synchronization. Plot retry timestamps; vertical spikes indicate missing or ineffective jitter.
- Check quota dimensions. Compare request, input-token, output-token, daily, model, and regional usage.
- Check headers. Confirm that a valid
Retry-Afteris parsed and that date values are not broken by clock skew. - Check queue and concurrency. Reduce in-flight work and drain backlog gradually.
- Check streaming state. Ensure a post-stream error is not replaying a side-effecting request.
After correcting the cause, restore capacity in stages. Keep the old limits available as a rollback, and compare success, latency, token usage, and duplicate-operation metrics before raising concurrency again.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhich provider or client is best for retry resilience?
There is no universal winner; evaluate the behavior that matters to your workload. A client with convenient automatic retries may still be unsuitable if you cannot bound the total deadline or observe individual attempts.
| Evaluation axis | Questions to ask |
|---|---|
| Error taxonomy | Does it expose provider codes, overload distinctions, and request IDs? |
Retry-After |
Is the header passed through and honored for both delay and HTTP-date forms? |
| SDK defaults | How many retries occur, which statuses qualify, and can settings be changed? |
| Quota visibility | Can you measure request, input-token, output-token, daily, model, and regional limits? |
| Streaming semantics | Can the client tell you whether output started and preserve error events? |
| Failover | Are alternate regions or models supported without violating residency or capability requirements? |
| Deadline control | Can one end-to-end deadline cover queueing, SDK work, backoff, and response reading? |
| Observability | Can you export attempt count, delay, status, code, model, region, and final outcome? |
Measure your own workload. Published cross-provider success-rate or latency figures do not establish how your prompt sizes, concurrency, regions, and streaming patterns will behave.
What is a practical rollout plan?
- Instrument first: add request IDs, provider codes, attempt numbers, token estimates, and deadlines before changing retry behavior.
- Classify conservatively: start with documented transient statuses and codes; keep permanent failures out.
- Set one budget: choose a physical-attempt ceiling and end-to-end deadline that include SDK retries.
- Add backoff and jitter: honor valid
Retry-After, then use a capped schedule based on measured workload behavior. - Limit concurrency: introduce queues, semaphores, and token-aware admission control.
- Test failure modes: inject 429, 408, transient 5xx, overload, malformed-request, cancellation, and post-stream errors.
- Ramp gradually: compare retry rate, queue age, deadline failures, token cost, and duplicate side effects at each traffic step.
Frequently Asked Questions
Should I retry a 429 immediately if no Retry-After header is present?
No. Treat a missing header as a reason to use your capped exponential schedule, not as permission for an immediate replay. Add jitter and let the operation deadline decide whether another attempt is still useful.
Can a request be throttled even when my requests-per-minute dashboard is below its limit?
Yes. Providers can enforce token, daily, model-specific, regional, or acceleration limits, and some return a slow-down signal during a sharp traffic increase. Inspect all exposed quota dimensions and smooth the ramp instead of watching RPM alone.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Is it safe to retry a tool call generated by an LLM?
Only when the tool operation is idempotent or has a deduplication mechanism. Use a stable operation key, persist the result, and make the tool reject or return the prior result for a duplicate key before enabling automatic replay.
How can I test overload handling without harming production traffic?
Run fault-injection tests against a stub or isolated account that returns realistic statuses, provider codes, headers, delayed responses, and stream error events. Assert maximum physical attempts, deadline compliance, cancellation, queue limits, and the absence of duplicate side effects.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
