Start by preserving the complete request and response, then classify the status code before changing your scraper. A 4xx usually means credentials, parameters, account state or target policy; a 5xx usually indicates provider or upstream conditions. For 429 and rate-limit 503 responses, honor Retry-After, reduce concurrency and retry with exponential backoff and jitter. For 403, CAPTCHA and access-denied pages, determine whether your provider account, proxy or the target site is responsible.
1. Capture evidence before you retry
Intermittent scraping failures become difficult to diagnose when each retry overwrites the original context. Record one structured event per request. Include:
- HTTP method and complete endpoint URL (without secrets).
- Query parameters or a redacted request body.
- Sanitized request headers, status code and response headers.
- The provider’s structured error object, if present.
- Total latency, retry count and timestamp.
- Proxy or session identifier, but not the proxy password.
- A short, non-sensitive response-body sample.
Never log API keys, cookies, authorization values or page credentials. Hash a session identifier if support needs correlation without seeing the actual value. Preserve the provider request or scrape ID whenever one is returned; Scrapfly, for example, exposes a scrape_id, a retryable flag and reject-code headers that are useful during escalation.
2. Validate the request shape and authentication
Check the URL and payload
Use an absolute target URL, encode query strings correctly, send valid JSON when the endpoint expects JSON, and set the matching Content-Type. Confirm required fields and parameter names against the provider’s current reference. A malformed body can produce either 400 or 422, depending on the service.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →#1 Best Overall
Check where the credential belongs
Providers do not use one universal authentication convention. Zyte’s reference uses HTTP Basic authentication with the API key as the username. Apify documents token-based authentication and structured 4xx errors. Verify the exact header, query parameter or Basic-auth placement required by your account, and load the secret from an environment variable rather than source code.
export SCRAPER_TOKEN='replace-me'
curl -i
-H "Authorization: Bearer $SCRAPER_TOKEN"
-H 'Content-Type: application/json'
-d '{"url":"https://example.com"}'
https://api.example.invalid/v1/scrapeReplace the endpoint and authentication format with your provider’s documented values. A 401 caused by an expired, misspelled or missing token will not be fixed by changing proxies or adding delays.
3. Classify the status code before changing code
| Status | Likely meaning | First action |
|---|---|---|
| 400 / 422 | Malformed JSON, invalid, missing or incompatible parameters | Compare the serialized request with the provider schema; validate types and required fields. |
| 401 | Missing, malformed or unknown API key/token | Verify secret source, auth placement and whether the key is active. |
| 403 | Account suspension, eligibility restriction or target-site denial | Separate provider account state from a target block by testing a harmless URL and inspecting the body. |
| 404 | Wrong endpoint, resource ID or target URL | Check the API path and that the target resource still exists. |
| 429 | Rate limit exceeded | Honor Retry-After, lower concurrency and apply jittered backoff. |
| 503 | Provider overload or rate limiting | Retry with backoff; use Retry-After when supplied. |
| 520 | Zyte temporary ban | Retry after a generous delay and inspect whether the target is blocking the current route. |
| 521 | Zyte permanent download error | Check request parameters and whether the domain is reachable; blind retries are unlikely to help. |
| 590–599 | Apify proxy or upstream diagnostics | Use the specific diagnostic: 593 DNS failure, 594 connection refused, 595 reset/timeout, 596 broken pipe, 597 upstream-auth failure, 599 generic upstream error. |
These meanings are provider-specific where noted. Do not assume that a status code has identical billing or retry semantics across services.
4. Retry throttling safely
Use server guidance first
If Retry-After is present, wait at least that long. Otherwise use exponential backoff with random jitter. A practical sequence starts at 500 ms and doubles (0.5, 1, 2, 4, 8 seconds), with a maximum delay appropriate to the provider. Cap retries for non-rate-limit failures so a bad URL or invalid credential does not create an endless loop.
import random
import time
import requests
def get_with_backoff(url, token, attempts=6):
for attempt in range(attempts):
response = requests.get(
url,
headers={"Authorization": f"Bearer {token}"},
timeout=60,
)
if response.status_code not in (429, 503):
return response
retry_after = response.headers.get("Retry-After")
if retry_after and retry_after.isdigit():
delay = float(retry_after)
else:
delay = min(60.0, 0.5 * (2 ** attempt))
delay += random.uniform(0, delay * 0.25)
time.sleep(delay)
raise RuntimeError("rate-limited after all retry attempts")
Do not retry 401, 400 or 422 unchanged. Fix the request first. For 403, retry only when your provider identifies a transient account or network condition; repeatedly replaying a blocked target can worsen its reputation.
Control concurrency, not just delay
A fast single request can succeed while a worker pool triggers 429. Add a per-provider semaphore or token bucket, and measure requests per second, concurrent jobs and queue depth. Apify documents a default limit of 60 requests per second per resource and a global limit of 250,000 requests per minute; those figures apply to Apify’s documented API, not to every scraping service. Zyte documents 3,000 requests per minute for Standard API keys, with separate website and account limits.
5. Tell a target block from an API failure
Compare browser and API responses
Open the same URL in a normal browser and through the API at roughly the same time. Look for CAPTCHA text, “access denied,” challenge scripts, unusual redirects, a much shorter HTML body or a provider-generated error object. Some sites deliberately serve different content to browsers and non-browser clients, so an API 200 can still contain a block page.
Test a controlled session
Use a stable session when login cookies, carts or multi-step navigation must persist. Rotate IPs when the evidence points to IP reputation rather than account state. Keep other variables constant while testing: changing user agent, cookies, proxy country and concurrency simultaneously prevents you from identifying the cause.
Verify proxy connectivity
For Apify Proxy, its documentation recommends checking the proxy status page and https://api.apify.com/v2/browser-info/ to confirm connectivity and the observed IP. Datacenter proxies are generally easier to scale; residential routes may have different reputation, geography and session behavior. Apify documents session persistence of about 26 hours for datacenter sessions and around 30 minutes for residential sessions, so design cookie-dependent workflows around those limits.
6. Troubleshoot common failures by symptom
“It works in my browser but not in code”
- Check whether the browser sends cookies, a referer, a JavaScript-generated token or a different user agent.
- Confirm that your API request follows redirects and supports the target’s TLS requirements.
- Inspect the response body for a challenge rather than trusting the status alone.
- Use a browser-rendering mode when the page is assembled by JavaScript; a plain HTTP client cannot execute it.
403 with no provider error
First call a provider-controlled or known public URL. If that succeeds, the target is probably denying the route, user agent or session. If both fail, check account eligibility, suspension notices and proxy authentication.
Timeouts, resets and blank pages
Distinguish DNS failure (Apify 593), connection refusal (594), reset or timeout (595), broken pipe (596) and generic upstream failure (599). For a target that is merely slow, increase the provider’s page timeout and wait for a meaningful selector or network idle. For a consistently blank response, verify that the target is reachable outside the provider and that required JavaScript resources are not blocked.
520 or 521 from Zyte
Treat 520 as a temporary ban: wait, then retry with a controlled rate and, if supported, a different route. Treat 521 as a permanent download error until you have checked the domain, URL encoding, request parameters and DNS/reachability. Capture the exact Zyte error distinction in your logs so support can reproduce it.
Recommended Free Tools
Best Value
7. Make diagnostics useful to support
Send a minimal reproducible request, UTC timestamp, target domain, provider request or scrape ID, status, latency, retry count, sanitized headers and the first few hundred bytes of the body. Include the proxy/session identifier and the reject code or description when available. Explain what changed immediately before the failure and whether a normal browser can load the page. This is more actionable than a screenshot of a dashboard or a message that “the API is down.”
8. When you need rendered screenshots instead of raw scraping
If your actual requirement is a reliable visual capture, a browser-and-proxy stack may be unnecessary. ScreenshotNeo is the first screenshot API to try here because it removes consent banners, newsletter popups and chat widgets before capture, bills only clean shots, and has the lowest paid plan.
Or skip the browser setup
One GET request returns PNG, JPEG, WebP or PDF. The API accepts the same parameter names used by many screenshot services, and the response identifies outcomes with X-Page-Verdict and X-Billed headers.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
See the complete option list and response details in the ScreenshotNeo documentation. Cookie banners, popups and chat widgets are removed before the shot; bot checks, blank pages and failed loads are never billed. Its MCP server lets Claude, Cursor and other MCP clients call take_screenshot, get_page_info and capture_pdf. The Free plan includes 1,000 screenshots a month with no card, and paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errors9. Build an operational checklist
- Store redacted request and response evidence for every failure.
- Classify status and provider error type before retrying.
- Honor
Retry-After; add exponential backoff and jitter. - Limit concurrency per resource and account.
- Separate provider authentication failures from target anti-bot blocks.
- Keep sessions stable for cookies; rotate routes only when IP reputation is implicated.
- Track latency, status distribution, retry count, billed outcomes and proxy diagnostics.
- Escalate with a reproducible request and provider request ID.
Frequently Asked Questions
Should I retry every 5xx response forever?
No. Retry rate-limit 503 responses with server guidance and backoff, but cap retries for other 5xx responses and escalate persistent failures with request IDs and timestamps.
Why can a 200 response still be unusable?
The body may be a CAPTCHA, consent wall or access-denied document. Inspect body markers and page structure instead of treating HTTP 200 as proof that the intended content was returned.
Which limit should I use when planning worker capacity?
Use the exact provider and account limit documented for your service, then leave headroom. Apify’s published 60-per-second resource and 250,000-per-minute global figures are not universal defaults.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
