Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteSome links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
A timeout in a web scraping API is not necessarily one clock measuring one thing. The API may limit how long its server works on a request, while its browser renderer separately waits for JavaScript, a particular element, or a browser event. Your own HTTP client can also stop waiting first. To diagnose a timeout—or avoid returning a page before it is ready—identify which clock expired, then inspect the response status and body.
What does a scraping API timeout measure?
A timeout sets a limit on waiting, but the phases covered by that limit depend on the provider and endpoint. An API-level timeout may bound the provider’s work on a request; a browser-rendering wait controls when a rendered page is considered ready; and the HTTP client making the request may have its own deadline. Those limits are related, but they are not interchangeable.
For example, if your client gives up before the provider is expected to finish, your application can report a client-side timeout even while the provider is still processing the scrape. Conversely, a provider can return a response within the client’s deadline that contains incomplete page content because the browser did not wait for the target element you needed.
There is no universal timeout value or definition that applies to every scraping API. Check the current reference for the specific endpoint you use, including its units, default, allowed range, and what work the setting covers.
#1 Best Overall
How ScrapingBee separates request time from render readiness
ScrapingBee’s documented HTML API is a useful provider-specific example, not an industry standard. Its timeout parameter is measured in milliseconds, defaults to 140,000 ms, and accepts values from 1,000 to 140,000 ms. The documentation gives a 0.5-second margin of error and warns: “Changing it could have a negative impact on your success rate.” These are ScrapingBee’s published settings and guidance; another provider may define the parameter differently.
A longer API timeout does not necessarily make a JavaScript-rendered page more complete. ScrapingBee documents separate rendering controls:
waitpauses rendering for a fixed duration, from 0 to 35,000 ms.wait_forwaits for a CSS or XPath selector.wait_browserwaits for a browser condition.
ScrapingBee notes that rendered HTML can arrive before some elements have rendered. If the data you need appears asynchronously, waiting for a known element or relevant browser condition is often a more targeted fix than simply increasing the overall request timeout. A fixed delay can help when readiness cannot be expressed as a condition, but it may waste time on fast pages and still be too short on slow ones.
What happens when a scraping API times out?
The result depends on where the request stops and how the provider represents the failure. You might see a client-library exception, a provider response with an error status and explanatory body, or a status that reflects the provider’s mapping rather than the target site’s original response. Do not infer the cause from a status code alone.
ScrapingBee documents that its default status mapping can convert many target errors into a provider-side 500. Its response body may contain the reason for that 500. Therefore, a returned 500 does not by itself prove that the target site returned HTTP 500; read the response body and check the provider’s status-mapping behavior.
ScrapingBee also documents a transparent_status_code=true option that changes status mapping. The trade-off matters: its documentation says transparent status mode disables the provider’s retry behavior, and the setting has billing implications. Do not enable it as a generic timeout fix. First decide whether exposing the target’s status is worth changing retries and billing behavior for your use case.
Timeout labels and codes vary by provider. Oxylabs’ company guide identifies code 524 as timeout/service unavailable, but that example does not establish a shared code across scraping APIs. Check the current documentation for the provider and endpoint that produced the response.
How long should you wait for a scraping API?
There is no generally correct timeout to copy across providers. Set a deadline based on the endpoint’s documented limits and the needs of your application, then make sure the client is allowed to wait long enough for that request to complete. If the client deadline is shorter than the time the provider may reasonably need, the client can abandon the request prematurely. The available provider examples do not establish a universal client-to-provider timeout ratio.
When a response arrives but the page content is incomplete, distinguish a readiness problem from a slow request:
- Identify the missing content. Determine whether it is present in the initial HTML or created or loaded by JavaScript.
- If it is rendered asynchronously, wait for the relevant condition. Use a selector or browser event supported by your provider when possible.
- Use a fixed render delay only when appropriate. Choose a duration that fits the page and the provider’s allowed range; a delay is not a guarantee that every element has appeared.
- Adjust the API request deadline only if processing itself needs more time. Stay within the provider’s documented bounds.
- Check the body and status mapping. Establish whether the failure came from the target, provider, or client before changing configuration.
Retries: useful for transient failures, not for every timeout
Retries can recover from temporary transport or provider failures, but they do not fix a deterministic target response or an incorrect readiness condition. Use bounded retries with backoff in your client code, and avoid retrying indefinitely. Before retrying, determine whether the request failed transiently or simply returned content that was not ready.
Retry behavior is provider- and client-specific. ScrapingBee documents retries for failed scrapes by default, but also documents that transparent status mode disables that behavior. Separately, its CLI documentation specifies three retry attempts by default for transient 5xx and connection errors, with exponential backoff multiplier 2 and documented delays of 2, 4, and 8 seconds. Those CLI defaults should not be assumed to apply to every ScrapingBee API client, or to other providers.
Recommended Free Tools
In your own integration, cap attempts and total elapsed time. Avoid stacking an aggressive client retry loop on top of a provider’s automatic retries without accounting for the combined attempts and wait. For non-idempotent operations, confirm the provider’s behavior before retrying; the cited documentation here does not establish a universal rule about duplicate request handling.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshoot a timeout or incomplete scrape
- Your client reports a timeout, but the provider may still be working. Compare the client’s deadline with the provider’s documented request limit and processing needs. Increase the client allowance if it is ending the wait too soon; do not assume this changes the provider’s render readiness.
- The API responds, but expected fields are missing. Check whether the target loads those fields through JavaScript. Add a selector or browser-condition wait if supported, rather than extending the API deadline alone.
- You receive a 500. Read the response body and consult the provider’s status mapping. For ScrapingBee, the default mapping can turn many target errors into provider-side 500 responses.
- You want the target’s status code surfaced directly. Review the provider’s transparent-status option and its consequences first. ScrapingBee documents that its option disables retry behavior and has billing implications.
- A retry loop keeps failing. Stop and distinguish transient connection or provider errors from a repeatable target error, an invalid request, or a page that never meets the requested readiness condition. Bound retries and use backoff.
- You see code 524. Oxylabs’ guide uses it for timeout/service unavailable. Treat that as an Oxylabs example, not a universal scraping API code, and check the current provider reference.
Or skip the browser setup
If your task is to capture a website screenshot rather than extract scraped HTML, ScreenshotNeo provides a one-request screenshot API. Its request returns an image or PDF; that is a different output from a general-purpose scraping API. For a simple image capture, start with this cURL request (replace the example URL with the page you need):
ScreenshotNeo API documentation
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpScreenshotNeo removes cookie and consent banners, newsletter popups, and chat widgets before capture; those steps can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and each response identifies the page verdict and billing status in headers. Its MCP server offers screenshot and PDF tools for AI agents using Claude, Cursor, or another MCP client. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 screenshots. See ScreenshotNeo’s free sign-up to get started.
What to check before changing a timeout
- Confirm the parameter belongs to the endpoint you are calling, and verify its unit, default, minimum, and maximum in the live reference.
- Separate the provider’s request deadline from browser rendering waits and your client’s own deadline.
- Inspect both response status and body, accounting for provider-specific status remapping.
- Use retries for transient failures with bounded attempts and backoff, not as a substitute for diagnosing slow targets or missing readiness signals.
- Check timeout and failure billing rules for the exact service and plan you use; policies are not established across providers by the examples above.
Frequently Asked Questions
Does a timeout mean the target website is down?
No. A timeout only establishes that a particular request did not finish within a particular deadline. The target may be slow, the provider may still be processing, or the client may have stopped waiting; the status and response body help distinguish these cases.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallIs 524 the standard timeout code for scraping APIs?
No. Oxylabs’ guide identifies 524 as timeout/service unavailable for its context. The example does not establish a standard shared by other providers.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
