The reliable way to improve production scraping success is to optimize valid, schema-checked records per attempted record—not request volume. Establish a per-target baseline, find the concurrency and delay each site tolerates, classify failures before retrying, and measure extraction quality separately from HTTP completion. There is no universal production success-rate benchmark; your denominator, target, route, and time window must be explicit.
Define “success rate” before changing the crawler
Choose a metric that represents data your application can use. A practical definition is:
success rate = valid expected records ÷ attempted records for a named target, route set, and time window.
“Valid” should mean that the response was parsed, required fields passed schema validation, and the record met freshness or business rules. A 200 response containing a bot page is not a successful extraction. Likewise, a completed request that produced no expected item should remain a failure in the application metric.
#1 Best Overall
Keep separate counters for:
- requests attempted and completed;
- HTTP status classes, including 2xx, 4xx, and 5xx;
- connection, DNS, TLS, and timeout exceptions;
- retries and final outcomes;
- parse success and schema-validation success;
- freshness, duplicate rate, and records per response.
Report each metric by host, route, status, worker pool, and time window. This prevents a healthy API endpoint from hiding a failing JavaScript route or an increase in technically successful but unusable pages.
Start with permission and the target’s documented access path
Check robots.txt and published terms
Review robots.txt, terms, authentication requirements, and any published crawling guidance before tuning load. Scrapy’s RobotsTxtMiddleware can filter forbidden requests when enabled. However, Scrapy does not automatically apply robots.txt Crawl-delay or Request-rate directives, so convert applicable guidance into explicit delay and concurrency settings.
Prefer an API, export, or search endpoint
If the site offers an official API, bulk export, or documented search endpoint, evaluate it first. Scrapy’s optimization guidance notes that these paths can be faster for your crawler and cheaper for the site than downloading page after page. They may also provide more stable schemas and clearer authentication than rendered HTML.
Use page crawling only for data that is not available through a permitted, documented path. Do not treat a missing API as permission to evade access controls.
Build a per-host baseline
Before increasing concurrency, run a representative workload at a conservative rate. Store measurements with a timestamp and deployment version so that a change in target behavior is not confused with a code change.
| Signal | What it tells you | Useful breakdown |
|---|---|---|
| Attempt count | Denominator for your rate | Host, route, time window |
| Status counts | Target-side responses and access signals | Exact code and status class |
| Latency | Load, route, and origin behavior | Median, tail percentile, route |
| Retries | Transient failure and retry pressure | Reason, attempt number, final result |
| Parse validity | Whether returned content is useful | Selector, schema error, record type |
| Freshness | Whether data meets its age requirement | Source timestamp and ingestion time |
Keep target responses distinct from crawler-side exceptions. A timeout is not a 504, and a 403 returned by the target is not the same as a local authentication configuration error.
Find the concurrency the target tolerates
There is no safe universal concurrency number. As Scrapy’s documentation puts it, “The limit that matters, though, is the one the target website tolerates.” Start low, increase gradually, and watch 429 responses, 503 responses, known access-control pages, retry counts, and latency.
- Choose a conservative per-host concurrency and delay.
- Run long enough to observe normal latency and status distributions, not just a short burst.
- Increase one variable at a time in small steps.
- Record the point at which rate limits, overload responses, or tail latency rise materially.
- Back off to the last stable setting and make that your operating range.
Target concurrency is an average goal, not a hard instantaneous cap. A queue, DNS pool, connection reuse, or several workers can create short bursts above the average. Enforce explicit per-host limits if bursts themselves cause failures.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteUse adaptive throttling instead of a fixed delay
Scrapy AutoThrottle adjusts delay from response latency and a target concurrency. It averages the new delay with the previous delay, respects configured minimum and maximum bounds, and does not allow a non-200 response’s latency to reduce the delay. Those rules make it useful when a site’s capacity changes during the day.
Configure a floor that avoids aggressive bursts and a ceiling that prevents a stalled target from consuming workers indefinitely. Set the target concurrency as an average objective, then validate it against status and extraction metrics. AutoThrottle cannot identify a bot page as valid data; pair it with content checks and schema validation.
Retry only failures that are plausibly transient
Set a finite budget
Scrapy’s documented default is two retries after the initial download, and its retry middleware includes 429 and selected 5xx responses. That is a framework setting, not an industry success-rate statistic. Verify the defaults for the exact Scrapy version you deploy.
For every retry, record the reason, delay, attempt number, and final outcome. Bound retries per request and per job. Exponential backoff with jitter prevents a fleet of workers from retrying simultaneously.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Do not retry access-control failures indefinitely
Repeated 429 responses, bot challenges, or policy-related 403 responses are signals to reduce load or fix access and permission issues. Retrying harder increases traffic while lowering useful yield. A response that is consistently an access-control page should be classified and surfaced for investigation, not hidden in a retry loop.
Classify errors before changing selectors or rate limits
4xx responses
- 404: verify URL construction, pagination, deleted content, and stale links.
- 401 or 403: check authentication, authorization, terms, and whether the route is intended for your client.
- 429: reduce concurrency, increase delay, honor published limits, and inspect whether several workers share one rate budget.
Cloudflare notes that 4xx crawl errors can result from missing pages or malformed links. Do not assume every 4xx is a temporary network problem.
5xx responses
Cloudflare also notes that 5xx errors may originate at Cloudflare or at the origin server. Compare response headers and timing across routes, check the target’s origin health when you control it, and retain enough request context to distinguish intermediary failures from application failures. Back off during overload rather than multiplying requests.
Timeouts and connection failures
Separate connect, TLS, read, and overall deadline failures. A short connect timeout can indicate DNS or network trouble; a long read timeout may indicate a slow route or an overloaded origin. Increase a timeout only when measurements show that legitimate responses need more time; otherwise, you can hold workers hostage and worsen queueing.
Validate content, not just transport
Build a content-level classifier for empty pages, login screens, challenge pages, and unexpected templates. Require stable identifiers and types before accepting a record. Validate required fields, data types, ranges, and source timestamps. Count each rejection reason.
For rendered pages, wait for a selector that proves the data is present rather than sleeping for an arbitrary duration. For paginated routes, detect repeated cursors and duplicate pages. Store a sample of rejected bodies or hashes under your privacy and retention rules so an operator can replay the parser against the actual failure.
Remove local bottlenecks and repeated work
A target may be healthy while your crawler is constrained by its own scheduler, CPU, memory, storage, DNS resolver, connection pool, or parser. Instrument queue time separately from network latency. Watch worker utilization, open connections, garbage collection, disk throughput, and database write latency.
Use caching during development to avoid repeatedly downloading the same pages. Narrow and reuse selectors to reduce parsing work. In production, deduplicate URLs, checkpoint progress, and make writes idempotent so a retry cannot create duplicate records. Keep a replayable request and parser version for every accepted batch.
Recommended Free Tools
Make production changes safely
- Define the valid-record denominator and acceptance schema.
- Capture a baseline by host and route.
- Confirm permission, robots.txt behavior, and documented access paths.
- Enable bounded retries with reason codes and backoff.
- Roll out concurrency or AutoThrottle changes to a small worker pool.
- Compare valid-record rate, 429/503 rate, latency tails, retry volume, and freshness against the baseline.
- Promote only if useful yield improves without violating target limits.
- Keep an automatic rollback threshold for access errors, queue growth, or schema failures.
Alert on changes in valid-record rate and failure composition, not merely on total request volume. A lower request rate can be an improvement if it produces more accepted, fresher records.
Choosing an access model at scale
Compare an official API or export, a self-managed crawler, and a managed extraction service against the workload rather than against marketing claims.
| Axis | Official API/export | Self-managed crawler | Managed service |
|---|---|---|---|
| Permission | Usually explicit and documented | You must verify route and policy | Verify how it obtains and documents access |
| Coverage | Limited to published resources | Broad, subject to target behavior | Depends on documented target coverage |
| Rendering | Often structured; confirm endpoint | You operate browser/rendering capacity | Confirm JavaScript and session support |
| Failure control | Provider-specific limits | Full retry, backoff, and queue control | Depends on service controls and logs |
| Validation | You still validate schemas | Parser and schema are yours | Confirm whether parsed output is inspectable |
| Observability | API metrics plus your telemetry | Full request and replay ownership | Check status detail, replay, and retention |
| Cost | Measure cost per valid record | Engineering and infrastructure burden | Usage fees plus integration and lock-in |
Make a workload-specific comparison using cost per valid record, latency, session requirements, observability, and maintenance—not request price alone.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Or skip the browser setup
When your pipeline needs rendered screenshots or PDFs as evidence, ScreenshotNeo provides a single request instead of maintaining browser workers. It accepts consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be disabled. Only clean shots are billed: bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and response headers identify the page verdict and billing result.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Its MCP server exposes take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients. Features include full-page lazy-image loading, CSS-selector element capture, device presets and custom viewports, dark mode, retina scale, PDF controls, custom CSS and JavaScript, clicks, selector or network-idle waits, request blocking, headers, cookies, user agents, Authorization, timezone and geolocation, transparent backgrounds, resizing, TTL caching, signed links, asynchronous webhooks, bulk capture for 100 URLs per call, usage data, and an OpenAPI specification. Common screenshot-API parameter names also work.
cURL:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);See the ScreenshotNeo documentation for request options. The Free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. Sign up free to try it.
Troubleshooting checklist
429s rose after a deployment
Compare per-host concurrency and shared IP or credential pools with the baseline. Reduce concurrency, increase delay, and inspect whether a new route multiplied requests.
Requests are 200 but valid records fell
Sample bodies and hashes. Look for challenge, login, empty, or changed templates; then update content classification or the parser rather than adding retries.
Free tools Windows power users keep installed
One-click scans. No signup required.
503s persist only on one route
Compare route latency and origin behavior. The failure may be route-specific or origin-side; isolate it before changing global crawler settings.
Best Value
Retries consume the queue
Group failures by reason, enforce a retry budget, add jittered backoff, and stop retrying repeated policy or challenge responses.
Throughput is low despite healthy targets
Inspect local queue time, CPU, memory, DNS, connection pools, storage, and parser duration. Cache development downloads and remove duplicate URL work.
FAQ
What is a good scraping success rate?
No universal production benchmark is established. Define valid records, attempts, target, route, and time window, then set a baseline and improve it without exceeding the target’s tolerated load.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteShould I always use a headless browser?
No. Prefer a documented API or export when available. Use rendering only when the required data is delivered through browser execution or the workload specifically needs visual output.
Is AutoThrottle a hard concurrency limit?
No. It targets an average concurrency and adjusts delay from latency while respecting configured bounds. Enforce explicit limits when instantaneous bursts matter.
Frequently Asked Questions
How should I compare a managed scraper with my crawler?
Compare cost per valid record, documented access, target coverage, rendering and session needs, 429/5xx handling, schema validation, replay and observability, latency, and maintenance for your own workload.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →
