Estimating Scraping Volume and API Usage: Requests, Bandwidth, Limits, and Retries

Estimate scraping demand by counting every request-producing action, measuring real response sizes, and then checking each service limit separately. A useful baseline is:

Total requests per run = detail pages + index pages + pagination + authentication and metadata calls + exports + expected retries. Multiply that by targets and scheduled runs to get daily volume. Estimate bandwidth from measured response bytes, then account for headers, redirects, retries, and exports. Finally, compare request, token, point, concurrency, and billing limits; the first limit you exhaust determines practical capacity.

#	Preview	Product	Price
1		AI Pricing @ Work: The Playbook for Pricing Usage-Based AI Products That Make Money on Your Best...	$39.14	Buy on Amazon

Start with request-producing units

A “page” is not a reliable usage unit. One product page may require one API call, while a listing endpoint can require several paginated calls and a separate metadata request. Hosted extractors may bill per successful result rather than per HTTP request. Define the unit your target actually counts before doing arithmetic.

Classify every call

• Index or list calls: category pages, search results, feeds, and collection endpoints.
• Detail calls: one URL or record per request.
• Pagination calls: every cursor, page number, or “next” request after the first response.
• Authentication and metadata: token refreshes, schema discovery, capability checks, and health calls.
• Exports: dataset downloads, report generation, or result-file retrieval.
• Retries: each retry is another request and can also repeat response bytes.

For a browser crawler, also count subresource requests if your provider bills them: HTML, scripts, stylesheets, images, fonts, and API calls made by the page. A service that bills one rendered capture treats those differently from a service that bills network requests.

#1 Best Overall

AI Pricing @ Work: The Playbook for Pricing Usage-Based AI Products That Make Money on Your Best Customers (Fast Track ◆ Slow Track)

$39.14

Buy on Amazon

The core formulas

Requests per run

For one pass over a target set:

requests_per_run = targets × (index_calls + detail_calls + pagination_calls + metadata_calls + export_calls) + expected_retries

If different targets have different page mixes, calculate each class separately and add the results rather than using one average that hides outliers.

Daily requests

daily_requests = requests_per_run × scheduled_runs_per_day

For a weekly job, divide the weekly total by seven only when comparing with a daily quota. Do not use that average to justify bursts: a job that runs 20,000 calls in one hour still needs to fit the one-hour and per-second windows.

Bandwidth

Measure average response bytes for each call class. A practical estimate is:

daily_bytes ≈ Σ(requests_by_class × average_response_bytes_by_class) + request_headers + response_headers + redirects + retries + export_bytes

Use captured wire sizes, not the uncompressed HTML length. Compression, transfer encoding, redirects, cookies, and TLS framing can change network usage. If your provider reports compressed transfer, use that same basis in your estimate.

Concurrency and rate

Concurrency is the maximum number of in-flight requests, not the daily total. A rough sustained rate is daily_requests ÷ 86,400 requests per second, but burst windows and concurrency limits must be checked separately. A low daily average can still trigger a per-second or simultaneous-request limit.

Measure a representative sample before scaling

1. Define scope: record domains, URL patterns, resources or records, refresh frequency, and whether exports are included.
2. Run a small sample: include ordinary, large, empty, redirected, and error-prone targets. Record status code, response bytes, latency, pagination depth, and retry count.
3. Separate classes: keep list, detail, pagination, metadata, and export measurements distinct. A single mean can understate large detail pages.
4. Calculate percentiles: use p95 latency and a high response-size percentile for capacity planning; retain the mean for cost estimates.
5. Add a safety margin: increase the estimate for growth, retries, and unusually deep pagination. Document the margin instead of treating it as measured usage.
6. Reconcile with provider telemetry: compare your counters with rate-limit headers, usage dashboards, and billing records after a real run.

A sample is useful only if it reflects production behavior. Sampling ten small pages from a site whose catalog contains large, JavaScript-heavy pages will produce a falsely low bandwidth estimate.

Worked example: a paginated catalog

Suppose an illustrative job processes 2,000 products. Each run fetches 20 category pages, one detail request per product, two metadata calls, and one export request. The measured retry rate is 3 percent of base requests.

With four scheduled runs per day, the estimate is 8,336 requests per day. If measured average transfer is 12 KB for index calls, 80 KB for details, 2 KB for metadata, and a 40 MB export, calculate each class separately; do not multiply one page size by all 2,084 requests. This example is a method, not a benchmark.

Apply limits in every dimension

Request and token limits

Some APIs separate request rate from token or payload rate. OpenAI documentation describes distinct request and token limits, project and organization scopes, reset headers, Retry-After, exponential backoff, and batching guidance. A request can fit the request-per-minute allowance while exceeding a token-per-minute allowance. Track both counters.

Authentication changes quotas

GitHub documents 60 requests per hour when unauthenticated and 5,000 requests per hour when authenticated. Its documentation also describes a secondary condition allowing no more than 100 concurrent requests. Verify which identity, endpoint, and authentication method your job uses; switching credentials can change the applicable quota.

Different windows can apply simultaneously

The Office for National Statistics documents 120 requests per 10 seconds, 200 requests per minute, and 15 requests per 10 seconds for high-demand assets. Exceeding a limit returns HTTP 429 and a Retry-After value. Your scheduler must satisfy the shortest window and the special asset limit, not just the one-minute number.

Read rate-limit headers

api.data.gov documents a default limit of 1,000 requests per hour. Its DEMO_KEY is limited to 30 requests per hour and 50 requests per day, and responses expose X-RateLimit-Limit and X-RateLimit-Remaining. Log those headers with each response when available; they reveal remaining capacity more accurately than a local counter when several workers share a key.

Retries, backoff, and pagination

Count retries before you launch

Model retries as an expected fraction of base calls:

expected_retries = base_requests × observed_retry_rate

Keep separate rates for timeouts, 429 responses, connection failures, and 5xx responses. A 3 percent average can conceal a single endpoint with a 20 percent failure rate.

Honor server timing

When a response includes Retry-After, wait at least that long. Otherwise use exponential backoff with jitter, cap the number of attempts, and cap total retry time. Never retry permanent client errors such as a consistently invalid URL or authentication failure.

Prevent pagination surprises

Record pages fetched per resource and stop on a stable end condition: an empty page, an explicit next cursor being absent, or a documented total being reached. Protect against a cursor loop by storing recently seen cursors. A changed sort order can otherwise make one run fetch the same records repeatedly.

Batch only when semantics allow

If an API supports batching, compare one batch request’s payload and token cost with the equivalent individual calls. Batching can reduce request pressure while increasing response size and retry blast radius. OpenAI documentation recommends the Batch API for collections that do not require immediate responses, because those jobs do not consume synchronous request rate limits in the same way.

Translate volume into a schedule

1. Convert the daily total to a rough sustained rate by dividing by 86,400.
2. Check every documented short window, such as 10-second and one-minute limits.
3. Set a worker cap below the provider’s concurrency ceiling, leaving room for health checks and retries.
4. Use a token bucket or leaky bucket so bursts cannot exceed the shortest window.
5. Reserve capacity for priority requests or emergency replays instead of running at 100 percent of the quota.

For multiple accounts or projects, calculate both per-credential and organization-wide totals. A credential rotation may move requests between keys without increasing an organization-level limit.

Self-hosted crawler or hosted scraping API?

Compare approaches using the unit that matters to your operation:

Scrapy.io documents synchronous and asynchronous scraper runs, polling, schedules, dataset downloads, and pay-per-result billing. If you use a hosted service, include polling requests and dataset exports in your usage model; they are not free merely because the extraction itself is asynchronous.

Browser captures and screenshot jobs

Browser rendering can multiply network activity because one URL may load many subresources. Decide whether your provider bills the rendered result, each underlying request, or both. Measure page-load time, asset count, transferred bytes, and failure rate with the exact viewport, user agent, and blocking rules you will use in production.

Or skip the browser setup

ScreenshotNeo is a website screenshot API and MCP server. It accepts consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be disabled. Only clean shots are billed: bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and the response identifies the outcome with X-Page-Verdict and X-Billed headers.

One GET request returns PNG, JPEG, WebP, or PDF. The API supports full-page captures with lazy images loaded, CSS-selector element captures, dark mode, device presets and custom viewports, retina scale, PDF paper and margin settings, custom CSS and JavaScript, pre-capture clicks, hidden selectors, selector/delay/network-idle waits, request and resource blocking, custom headers, cookies, user agents and Authorization, timezone and geolocation, transparent backgrounds, resizing, chosen TTL caching, signed image links, asynchronous jobs with signed webhooks, bulk capture of up to 100 URLs per call, usage reporting, and an OpenAPI specification. Parameter names used by other screenshot APIs also work.

cURL (see the ScreenshotNeo documentation):

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

Python:

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

Node.js:

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo includes an MCP server with take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients. The Free plan includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.

A runnable estimator

This small Python program keeps call classes separate and adds a retry margin. Replace the illustrative inputs with measurements from your sample.

from dataclasses import dataclass

@dataclass
class CallClass:
    name: str
    count_per_run: int
    avg_bytes: int

classes = [
    CallClass("index", 20, 12_000),
    CallClass("detail", 2_000, 80_000),
    CallClass("metadata", 2, 2_000),
    CallClass("export", 1, 40_000_000),
]
runs_per_day = 4
retry_rate = 0.03

base_requests = sum(c.count_per_run for c in classes)
expected_retries = round(base_requests * retry_rate)
requests_per_run = base_requests + expected_retries
daily_requests = requests_per_run * runs_per_day
bytes_per_run = sum(c.count_per_run * c.avg_bytes for c in classes)
daily_bytes = bytes_per_run * runs_per_day * (1 + retry_rate)

print(f"Base requests/run: {base_requests}")
print(f"Estimated retries/run: {expected_retries}")
print(f"Requests/run: {requests_per_run}")
print(f"Requests/day: {daily_requests}")
print(f"Approximate bytes/day: {daily_bytes:,.0f}")

The script estimates demand; it does not know a provider’s quota. Add checks for request windows, token usage, concurrency, and billing units before scheduling workers.

Troubleshooting common estimation failures

“Our dashboard shows more requests than our URL count.”

Check pagination, authentication refreshes, redirects, JavaScript subresources, polling, and retries. Instrument a request ID and call class so each extra request has a reason.

“We average under the limit but still receive 429.”

Your burst or concurrency is probably too high, or a stricter endpoint-specific window applies. Add a short-window limiter, reduce workers, and honor Retry-After.

“Bandwidth is far above the HTML size.”

Measure compressed wire bytes and include response headers, redirects, images, scripts, fonts, retries, and exports. Browser captures often fetch substantially more than the initial document.

“Retries made an outage worse.”

Unbounded immediate retries create a feedback loop. Use exponential backoff with jitter, a finite attempt count, a total-time cap, and a dead-letter queue for calls that need review.

“The estimate was accurate for one run but not the next.”

Compare page mix, pagination depth, response-size percentiles, and status-code distributions. Recalculate after catalog growth, authentication changes, endpoint migrations, or a new rendering configuration.

Quick Recap

Bestseller No. 1

AI Pricing @ Work: The Playbook for Pricing Usage-Based AI Products That Make Money on Your Best Customers (Fast Track ◆ Slow Track)

$39.14

Operational checklist

• Count list, detail, pagination, metadata, export, and retry requests.
• Measure response bytes and p95 latency by call class.
• Track request, token, point, concurrency, and billing units separately.
• Log rate-limit and retry headers, including Retry-After.
• Use backoff, jitter, attempt caps, and cursor-loop protection.
• Schedule against the shortest rate window, not only a daily average.
• Reserve quota for retries and urgent work.
• Reconcile local counters with provider usage after each production run.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.