What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Retry an LLM API call only when the provider indicates the failure may be temporary. For those failures, use exponential backoff with jitter, a strict attempt and time limit, and a service-wide retry budget where traffic is concurrent. A circuit breaker complements retries: it stops sending calls to a provider that keeps failing, then permits a small number of probes after a cooldown.
How should you retry LLM API calls?
Start by classifying the failure, not by repeating the request. Provider status codes and error bodies are not a universal contract: the same broad HTTP status can represent different conditions, and SDKs may retry automatically. Check the provider’s current error documentation, response body, and retry-related headers before defining policy.
As an Amazon Associate I earn from qualifying purchases.
- Usually not retryable unchanged: malformed requests, invalid credentials, permission failures, and billing or spend-limit conditions. Fix the request or account condition first.
- Potentially retryable: documented transient overload, throttling, network failures, and timeouts. Follow the provider’s guidance for the specific error subtype.
- Ambiguous outcomes: after a timeout, the provider may have processed the request even if the client never received the response. Consider duplicate cost or side effects before repeating it; use an idempotency feature if that endpoint offers one.
For example, Google’s Gemini troubleshooting guide identifies 429, 408, and 5xx errors as examples of transient failures to retry, while advising against retries for client errors such as 400, 402, or 403. Its error guidance also distinguishes cases that need a changed client deadline or a billing fix from failures appropriate for unchanged retries. These are Gemini-specific examples, not rules for every LLM API.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →What is exponential backoff, and how should you bound it?
Exponential backoff increases the wait after successive failed attempts instead of retrying immediately. Add random jitter so many clients do not retry together and recreate the load spike. When the provider supplies a retry hint such as Retry-After, or a provider-specific equivalent, honor it according to that error’s documented meaning.
#1 Best Overall
There is no universal LLM retry schedule. Set three bounds: a maximum number of attempts, a maximum delay, and a total elapsed-time budget. Choose the budget based on the caller: an interactive request generally needs a tighter deadline than a delay-tolerant background job. Stop when the caller’s deadline or retry budget is exhausted rather than letting retries run indefinitely.
Google’s Gemini troubleshooting page gives a concrete SDK example: its Python SDK documentation describes up to four automatic retries, an initial delay of approximately one second, and a maximum delay of 60 seconds. Treat those figures as that documented Python SDK example, not a prescription for other providers, languages, SDK versions, or application-level retries.
Should you retry a 429 or 503?
Only if the provider’s documentation and error details indicate a transient condition. A 429 can mean short-lived rate throttling, but it can also reflect a spend or account limit that will not clear after a brief pause. A 503 commonly indicates unavailability, but verify the provider’s own guidance and any retry hint.
Recommended Free Tools
Gemini API examples
Google’s Gemini troubleshooting guide gives 429 RESOURCE_EXHAUSTED and 503 UNAVAILABLE as retry examples and recommends exponential backoff with jitter and a maximum retry count. The Gemini API error reference says 429 rate-limit and 503 service-unavailable errors should wait and retry with exponential backoff; it also says 500 errors may be retried. For 504 deadline-exceeded errors, it directs users to examine or adjust the client deadline, while a payment-required condition requires adding credit or enabling auto-reload rather than retrying unchanged.
Rank #3
Gemini quotas are not a single request-per-minute number. Google describes requests per minute, input tokens per minute, and requests per day, with limits varying by model and usage tier. Limits apply per project rather than per API key and can change with tier and account status; check the project’s current limits in Google AI Studio and the Gemini rate-limit documentation rather than hard-coding a general value.
Claude API examples
Anthropic’s Claude API error documentation distinguishes 429 rate-limit errors, 500 internal errors, 504 timeouts, and 529 overload errors, and recommends exponential backoff for 500 errors. It also notes that some 429 spend-limit conditions may have no retry-after header and can persist until access resumes. Do not treat every 429 as a short rate-window throttle.
What does a circuit breaker do for API calls?
A retry assumes a call may succeed if attempted again after a suitable delay. A circuit breaker assumes repeated calls are unlikely to succeed right now and temporarily blocks them. Microsoft describes these as different patterns that can be combined: retry transient failures, but stop when the breaker is open or the failure is not transient. See the Azure Circuit Breaker pattern.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →- Closed: calls reach the provider and failures are counted over a configured window.
- Open: after the configured failure condition is met, calls are rejected quickly without reaching the provider.
- Half-open: after a cooldown, a limited number of probe calls are allowed. Success closes the breaker; failure reopens it and restarts the cooldown.
Thresholds, windows, cooldowns, and probe counts are workload-specific. Tune and validate them against traffic, user latency tolerance, provider behavior, and request cost; there is no established universal setting for LLM APIs. Make an open-circuit response distinguishable from a provider error so callers and operators know the call was intentionally blocked.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How do you keep retries from making an outage worse?
A per-request cap is not enough when many requests fail together. Each request may obey its own retry limit while their combined retries overwhelm a recovering provider or your own service. Microsoft’s guidance on transient faults recommends considering retry limits and aggregate retry budgets.
Quick Recap
- Account for retries at every layer. Add together SDK, client, and application-level attempts. If an SDK retries automatically, include those attempts and their delays in the outer request’s total budget; configure or disable retries where appropriate.
- Cap combined retry traffic. Use a service-wide retry budget, rate limit, or concurrency control. When it is exhausted, return a bounded failure or queue work if the user experience allows.
- Use the breaker for sustained failure. A breaker can prevent repeated doomed calls from consuming resources while the provider is unhealthy; allow only limited recovery probes.
- Observe the policy. Record attempt counts, final outcomes, latency, breaker state changes, and provider request identifiers when available. Breaker state-change events can help monitor component health.
- Protect against duplicate effects. Before retrying after uncertain timeouts, assess duplicate cost and side effects, and use endpoint-supported idempotency where available.
A practical implementation checklist
- Read the provider’s current documentation for the exact error code and structured error type. Inspect response headers and body for retry hints; do not assume all 429 responses mean the same thing.
- Separate documented transient overload, throttling, network, and timeout failures from permanent request, authentication, permission, and billing problems.
- For eligible failures, apply exponential backoff with jitter. Bound the attempts, individual delay, and total elapsed time to the caller’s latency budget.
- Calculate retry multiplication across SDK and application layers. Do not allow nested defaults to silently turn a small retry count into many provider calls.
- Set an aggregate retry budget or rate/concurrency control for bursts of failures. Decide whether budget exhaustion should fail promptly or queue delay-tolerant work.
- Place a circuit breaker around the provider call if persistent failures could cascade or waste resources. Define its failure window, threshold, cooldown, and limited half-open probes for your workload.
- Log enough to distinguish provider errors from local policy decisions, including breaker-open rejections and exhausted budgets.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
