Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
For a batch of public URLs, the most direct route is to send one Google PageSpeed Insights (PSI) API request per URL, with your own bounded concurrency, retries, and result handling. For a managed batch, Lighthouse CI’s psiCollectCron accepts URL arrays and lets you cap parallel URLs; for private sites use its Node mode instead of PSI. If you need checks from several geographic regions, Lighthouse Metrics API runs a check in each requested region. Whichever route you choose, repeat runs and compare like with like: concurrency speeds collection, but does not make Lighthouse scores more stable.
Choose the right way to fan out Lighthouse tests
There is no single Lighthouse endpoint that accepts a list of URLs and returns one bulk report. Google’s PageSpeed Insights API analyzes one URL per request. Bulk testing therefore means coordinating multiple requests yourself or using a collection service that does so. The right option depends on where pages can be reached, whether you need geographic checks, and how much scheduling and report handling you want to own.
| Approach | Where the test runs | Batch and concurrency | Best fit and constraints |
|---|---|---|---|
| Direct Google PSI API | Google-hosted runner | One URL per request; your client fans out requests | Small or custom batches of publicly reachable URLs. You must handle authentication, service quotas, backoff, and result storage. |
| Lighthouse CI, PSI collection | Google-hosted PSI runner | URL arrays; maxNumberOfParallelUrls controls concurrent URLs |
Repeatable collection as part of a Lighthouse CI workflow. URLs must be publicly accessible. The documented parallel-URL default is Infinity, so set a cap. |
| Lighthouse CI, Node mode | Your Node environment | Use the local collection workflow for targets your environment can reach | Private or otherwise non-public environments. It avoids PSI’s public-URL requirement, but it is not the same hosted runner as PSI. |
| Lighthouse Metrics API | Third-party regions | One check can request several regions; the service creates a run for each region | Geographic comparisons, monitors, and report retrieval. Bearer authentication is required, and rate limits can produce HTTP 429. |
PSI’s runPagespeed endpoint takes a URL and can select Lighthouse categories and a desktop or mobile strategy. Lighthouse CI documents numberOfRuns as defaulting to 5; choose repeated runs deliberately rather than treating a single result as definitive. Lighthouse Metrics supports optional device and Lighthouse-version selection, which helps control comparisons across its requested regions.
Run a public-URL batch with the PSI API
Use one request per URL, then limit the number in flight. The following Python example uses only the standard library. It takes URLs from a text file with one URL per line, requests the performance category and mobile strategy, retries transient HTTP failures with exponential delays, and writes successful response bodies as JSON files. Set a Google API key in GOOGLE_PSI_API_KEY if your API setup requires one; do not commit a real key to source control. The API quota and authentication configuration depend on your Google project, so check the project’s applicable settings rather than assuming a universal request allowance.
#1 Best Overall
Python: bounded parallel requests with retries
import concurrent.futures
import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request
ENDPOINT = "https://pagespeedonline.googleapis.com/pagespeedonline/v5/runPagespeed"
API_KEY = os.environ.get("GOOGLE_PSI_API_KEY")
CONCURRENCY = 3
MAX_ATTEMPTS = 4
with open("urls.txt", encoding="utf-8") as f:
urls = [line.strip() for line in f if line.strip() and not line.lstrip().startswith("#")]
def fetch_one(index, url):
params = {"url": url, "category": "performance", "strategy": "mobile"}
if API_KEY:
params["key"] = API_KEY
request_url = ENDPOINT + "?" + urllib.parse.urlencode(params)
for attempt in range(MAX_ATTEMPTS):
req = urllib.request.Request(request_url, headers={"Accept": "application/json"})
try:
with urllib.request.urlopen(req, timeout=120) as response:
payload = response.read()
result = json.loads(payload)
filename = f"result-{index:04d}.json"
with open(filename, "w", encoding="utf-8") as out:
json.dump(result, out, ensure_ascii=False, indent=2)
return url, "ok", filename
except urllib.error.HTTPError as exc:
# Retry throttling and server failures; other 4xx errors are usually input/config issues.
if exc.code not in (429, 500, 502, 503, 504) or attempt == MAX_ATTEMPTS - 1:
return url, f"HTTP {exc.code}", exc.read().decode("utf-8", "replace")
time.sleep(min(2 ** attempt, 30))
except (urllib.error.URLError, TimeoutError) as exc:
if attempt == MAX_ATTEMPTS - 1:
return url, "network/timeout error", str(exc)
time.sleep(min(2 ** attempt, 30))
with concurrent.futures.ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
futures = [pool.submit(fetch_one, i, url) for i, url in enumerate(urls, 1)]
for future in concurrent.futures.as_completed(futures):
url, status, detail = future.result()
print(status, url, detail)Start with a small concurrency value and increase it only when your quota, runner behavior, and collection window support it. Retries are intentionally limited to throttling, server-side errors, and network or timeout failures; repeatedly retrying a malformed URL or other permanent client error only wastes time. A successful HTTP response is not itself a performance pass: inspect the returned Lighthouse categories, audits, and requested strategy in each JSON response.
cURL: inspect one PSI response
Use this to validate an individual URL and parameter combination before launching a large batch. Replace the example URL and add your project’s API key if required by your setup.
curl -G "https://pagespeedonline.googleapis.com/pagespeedonline/v5/runPagespeed"
--data-urlencode "url=https://example.com/"
--data-urlencode "category=performance"
--data-urlencode "strategy=mobile"
--data-urlencode "key=$GOOGLE_PSI_API_KEY"Configure Lighthouse CI for collected URL batches
If you want Lighthouse CI to manage collection rather than writing a request fan-out loop, configure psiCollectCron.sites[i].urls with the URLs for a site and set maxNumberOfParallelUrls to an explicit, bounded number. Do not rely on its documented default of Infinity for a large batch: that can create a much larger burst than intended. Set numberOfRuns to the repeat count you need; its documented default is 5. The configuration also supports category arrays and mobile or desktop strategy selection.
Rank #2
Use PSI collection only for publicly reachable URLs. For an internal staging host, authenticated private page, or target unavailable to Google’s hosted runner, use Lighthouse CI’s Node method in an environment that can reach the site. The available documentation identifies Node mode as the appropriate private-environment method, but does not specify a single configuration template for every Lighthouse CI version; follow the configuration reference for the version installed in your project.
Make a batch operationally predictable
- Group pages by comparable purpose, such as templates or release-critical routes, so an error on one URL does not obscure results for the rest.
- Set a finite concurrency cap and run count. Larger batches and more repetitions consume more time and API capacity; concurrency changes throughput, not the validity of the scores.
- Keep the raw per-run results. A single aggregate can identify a regression, while individual runs help explain whether the change is consistent or driven by an outlier.
- Record the URL, strategy, device, Lighthouse version, region, and fetch time with each result. Do not combine observations from different dimensions as if they were one equivalent test.
Use regions for geographic comparisons
When the question is how a page performs from different locations, Lighthouse Metrics API offers a different model from PSI fan-out: POST /v1/lighthouse/checks accepts a URL and a regions array, and the service creates a run for each region. Optional device and Lighthouse-version settings help keep those regional runs more controlled. The API also supports monitors and report retrieval. Bearer authentication is required; a rate-limited request can return HTTP 429, so clients should handle throttling rather than treating it as a valid performance result.
Regional results answer a location-sensitive question; they should not be pooled with PSI output or local Node runs without labeling the runner and region. The service documentation identifies the API path and capabilities but does not supply a service base URL, current regional inventory, commercial limits, or report-retention period. Confirm those particulars with the service before relying on them for a production monitoring design.
Make repeated scores useful instead of noisy
Lighthouse scores vary between runs. Google’s Lighthouse variability guidance recommends aggregate values such as the median or 90th percentile rather than trusting a single observation. It states: “The median Lighthouse score of 5 runs is twice as stable as 1 run.” That is a stability comparison, not a guarantee that five runs eliminate noise or make scores comparable across different setups.
A practical comparison protocol
- Choose a consistent URL set, strategy, device, runner, and Lighthouse version for the comparison. If the test is regional, preserve each region as a separate dimension.
- Run multiple samples for each tested page and configuration. Use the median or a percentile for the headline comparison, and retain the individual readings for diagnosis.
- For a CI gate, select a representative aggregate such as the median rather than failing a build based solely on one unusually slow run. Set the threshold in the context of the same test setup.
- When a score shifts, inspect the raw run data and relevant audits before attributing the change to a code release. Check whether the runner, region, device, version, or fetch time changed too.
Do not average desktop and mobile measurements together, or treat separate geographic runs as repeated samples of the same condition. A score without its test dimensions is difficult to interpret later.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshooting parallel runs
HTTP 429 or throttled requests
Your request rate is being limited. Lower concurrency, add delayed retries with backoff, and check the applicable Google project quota or third-party service limit. Do not immediately resubmit the entire batch at the same rate.
Rank #4
A private URL fails in PSI collection
PSI’s hosted runner needs a publicly accessible URL. Use Lighthouse CI’s Node mode from an environment that can reach the private site instead of trying to make PSI access an internal host.
One URL returns an error while the rest finish
Keep failures associated with their input URLs rather than aborting the batch. Check URL validity and reachability, inspect the returned HTTP status and response body, and retry only transient failures. A batch is not complete merely because the client submitted all its requests.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteScores disagree between runs
That can be measurement variability, especially with a single sample. Compare multiple runs using a median or percentile, then verify that strategy, device, region, runner, and Lighthouse version match before interpreting the difference as a product change.
Best Value
- Used Book in Good Condition
CI creates an unexpected request burst
Set Lighthouse CI’s maxNumberOfParallelUrls explicitly. Its documented default is Infinity; a finite cap makes the load and collection rate more predictable.
Or skip the browser setup
ScreenshotNeo is a website screenshot API and MCP server, not a Lighthouse performance-testing API: it returns screenshots or PDFs, not Lighthouse scores or audits. If you also need clean visual captures in your workflow, a single GET request can produce one without setting up a browser. See the ScreenshotNeo API documentation.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpCookie banners, newsletter popups, and chat widgets are removed before capture; bot checks, blank pages, and failed loads are never billed. Its MCP server lets AI agents take screenshots. The free plan includes 1,000 screenshots a month with no card, and paid plans start at $5 for 3,000. Sign up for ScreenshotNeo’s free plan.
Recommended Free Tools
Frequently Asked Questions
Can Lighthouse CI run tests on URLs that are not public?
PSI collection needs publicly accessible URLs. For private targets, Lighthouse CI’s Node mode is the suitable route when run in an environment that can access them.
Should I compare Lighthouse scores from different regions?
Yes, when geography is the question—but keep each region labeled and compare like-for-like configurations rather than pooling regional results as equivalent samples.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
