The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →When a scraper starts receiving rate-limit errors or ban pages, reduce its load rather than trying to get around the block. There is no universally safe request rate: the limit that matters is what the target site tolerates. Scrapy’s AutoThrottle can adapt delays to response latency, but neither a longer delay nor any setting guarantees access or prevents a site from blocking a crawler.
Why a scraper gets blocked
A site may respond to a request rate it considers excessive with HTTP 429 or 503 responses, a ban page, retries, or worsening response times. Treat these as signals to back off, not as a reason to increase concurrency or evade the site’s limits. Scrapy’s optimization guidance says the target site’s tolerated rate is the relevant limit: Scrapy: Avoid getting banned.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
The Proxy Playbook: The Complete Guide to Proxy Servers: How to Source, Test, and Scale Residential,... | $29.95 | Buy on Amazon |
| 2 |
|
How to Host your own Web Server | $15.60 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
There is no evidence that slowing a crawler by a particular multiplier will reliably stop blocks. The useful question is whether the site’s published guidance and the crawler’s observed responses support the rate you are using.
Free tools Windows power users keep installed
One-click scans. No signup required.
Check the site’s rules and preferred data access
Read robots.txt and site guidance
Check the site’s robots.txt and any applicable terms or crawler guidance before collecting pages. Google explains that robots.txt is primarily for managing crawler traffic; it is not access control, and it cannot secure private information: Google Search Central: Introduction to robots.txt.
Scrapy does not automatically apply robots.txt Crawl-delay or Request-rate directives. If the site’s guidance includes them, translate those limits into your Scrapy delay and concurrency settings rather than assuming the directives are enforced for you. Follow applicable site terms as well as crawler directives.
Look for an API or export
If the site offers an API, bulk export, or search endpoint for the data you need, use that permitted route where it fits. It may be more efficient for you and less costly for the site than crawling pages. Check its permitted use, coverage, rate limits, freshness, and implementation effort before choosing; availability varies by site. See Scrapy’s guidance on avoiding bans.
Choose a Scrapy throttling approach
Use fixed delay and concurrency limits when you need predictable bounds
DOWNLOAD_DELAY and concurrency settings let you limit how quickly requests are sent. Set them in line with the target site’s guidance and adjust based on responses. A low fixed delay combined with a concurrency cap can still allow bursts; if errors return quickly, the crawler may send requests faster than intended. Scrapy discusses this limitation in its optimization guidance.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Use AutoThrottle when request latency should influence pacing
Scrapy AutoThrottle adjusts the delay for each download slot using observed response latency and a target average concurrency. It starts from AUTOTHROTTLE_START_DELAY, calculates a target delay from latency divided by target concurrency, averages that with the previous delay, then keeps the result within the configured minimum and maximum. Non-200 response latencies cannot make the delay decrease. The behavior and settings are documented in Scrapy AutoThrottle.
Rank #2
The Scrapy documentation lists defaults of a 5.0-second starting delay, a 60.0-second maximum delay, and a target concurrency of 1.0. These are software defaults, not measured safe rates or recommendations for every site. Choose values according to the site’s limits and your crawler’s responses.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Respond to rate-limit and server errors
HTTP 429 means the server is rate-limiting requests. RFC 6585 says a response may include Retry-After to indicate when a client should try again. Respect that guidance, reduce the request load, and avoid treating a rate limit as a challenge to bypass: RFC 6585, section 4.
Also watch for 503 responses, ban pages, repeated retries, and rising latency. Scrapy’s AutoThrottle documentation notes that errors may result from a high request rate and that slowing down makes more sense in that case: Scrapy AutoThrottle.
- If 429 responses appear, honor any
Retry-Aftervalue and lower the request rate. - If 503 responses, ban pages, or retries rise, reduce load and reassess concurrency and delay.
- If latency increases, allow more time between requests; AutoThrottle can adjust delay based on latency, within its configured bounds.
- If the site continues to reject requests, stop crawling and check its guidance or use an authorized data-access route.
How to tell whether your settings are appropriate
There is no universal delay that makes scraping acceptable to every site. Evaluate the crawler against the target’s published rules and its response signals. A configuration that produces fewer errors on one site is not proof that the same rate is appropriate elsewhere.
Quick Recap
- Confirm the crawler is limited by the site’s guidance, including any delay or request-rate instructions.
- Check whether 429 or 503 responses, ban pages, retries, or response latency are increasing.
- Account for the possibility that a fixed delay and concurrency cap can still create bursts.
- Prefer an API, export, or search endpoint when the site provides a suitable permitted option.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
