To capture many URLs reliably, put browser work behind a queue: an API or producer validates each request and submits a job, workers use Puppeteer to navigate and capture, and a result store or delivery step records the outcome. The queue lets you control concurrency and retry failures, but does not by itself guarantee durability or reliability; those depend on your queue backend, worker limits, timeout and retry policies, and result handling.
How do I queue screenshots for many URLs?
Separate request intake from browser execution. A practical service has four parts:
As an Amazon Associate I earn from qualifying purchases.
- Producer: accepts a URL and capture options such as viewport and output format; validates the input; and submits one job per URL.
- Queue: holds pending jobs so the producer does not have to run a browser for each request inline.
- Worker: takes a job, launches or reuses a controlled browser, opens a page, navigates, waits for the chosen readiness signal, and captures the page.
- Result handling: stores or delivers the image and records whether the job succeeded or failed. Define how long results remain available and how callers retrieve them.
Puppeteer’s capture method is Page.screenshot(). Its guide demonstrates navigation followed by a wait such as networkidle2, and also explains element screenshots. Choose readiness based on the target page: network idle can be unsuitable for pages with persistent connections or ongoing requests, while a selector wait or a deliberate delay may better match the content you need. See the Puppeteer screenshots guide and Page.screenshot API.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Submit a batch with BullMQ
BullMQ’s Queue.addBulk() accepts an array of jobs and may be faster than adding jobs one at a time. Give each job the URL and the options its worker needs; keep credentials and other sensitive values out of logs and public job status responses. Refer to the BullMQ Queue API for the method’s current details.
import { Queue } from 'bullmq';
const screenshots = new Queue('screenshots', {
connection: { host: '127.0.0.1', port: 6379 },
});
const requests = [
{ url: 'https://example.com', viewport: { width: 1440, height: 900 } },
{ url: 'https://example.org', viewport: { width: 1280, height: 800 } },
];
const jobs = await screenshots.addBulk(
requests.map((capture) => ({
name: 'capture-url',
data: { ...capture, format: 'png' },
})),
);
console.log(jobs.map((job) => job.id));This illustrates batch submission, not a complete service. Configure the Redis connection for your deployment, create workers that process the same queue, and define how job IDs map to status and results. Bulk submission reduces the producer-side work of adding jobs sequentially; it does not establish how many browsers can run at once.
Capture in a worker
A worker can use Puppeteer’s browser and page APIs to navigate, wait, and capture. The exact worker configuration depends on how you deploy browsers and the queue. Keep navigation, screenshot, and total-job timeouts explicit, and always close pages or return them to a managed pool when the job ends.
import { Worker } from 'bullmq';
import puppeteer from 'puppeteer';
const worker = new Worker('screenshots', async (job) => {
const browser = await puppeteer.launch({ headless: true });
try {
const page = await browser.newPage();
await page.setViewport(job.data.viewport);
await page.goto(job.data.url, {
waitUntil: 'networkidle2',
timeout: 30_000,
});
const image = await page.screenshot({
type: job.data.format ?? 'png',
fullPage: true,
});
// Store or deliver `image` here, then return a result reference.
return { captured: true, bytes: image.length };
} finally {
await browser.close();
}
}, {
connection: { host: '127.0.0.1', port: 6379 },
concurrency: 2,
});The sample uses a simple browser lifecycle so the cleanup path is visible. In production, decide whether workers should launch a browser per job or reuse browser processes, how to isolate pages, and how to recover from browser crashes. A fixed concurrency value is an example setting, not a general capacity recommendation: memory use, page complexity, browser configuration, and machine limits all affect safe worker capacity.
What should a bulk capture service handle?
Validation and safety
- Accept only supported URL schemes, normally HTTP or HTTPS, and reject malformed input before it reaches a browser.
- Protect the service from server-side request forgery: decide which hosts, IP ranges, redirects, and DNS results workers may access. Revalidate destinations after redirects where appropriate.
- Set maximum URL length, batch size, navigation duration, and output size to prevent a single request from consuming disproportionate resources.
- Validate capture options against an allowlist rather than passing arbitrary data into browser or storage configuration.
Retries, idempotency, and results
- Classify failures before retrying. A transient navigation or infrastructure error may merit a retry; an invalid URL or consistently blocked page may not.
- Use bounded retries and record the final error so callers can distinguish a failed capture from a job that is still pending.
- Make duplicate submissions safe. An idempotency key or deduplication policy can prevent clients’ repeated requests from creating unintended duplicate work.
- Persist outputs or upload them to object storage, then return a stable result reference. Set retention and access rules deliberately; a queue is not a long-term image store.
- Define what happens if capture succeeds but result storage fails, or if a worker exits after saving an image but before acknowledging the job.
Capacity and durability
Worker concurrency, queue length, and provider rate limits are different constraints. A queue can smooth bursts, but it cannot create browser capacity; workers still need limits that fit available resources. For work that must survive process restarts, use a durable external queue and configure recovery behavior rather than relying on in-process memory. ScreenshotOne recommends retries, respecting request buckets, and a durable Redis/BullMQ- or SQS-style queue when jobs must survive restarts. Its guide also cautions that a request bucket describes how many requests can be started in a time bucket, not how many screenshot renders can run simultaneously. See its bulk screenshots guide.
How should I choose readiness and screenshot options?
The correct capture settings depend on what the caller needs from each URL. Treat them as per-job data, validate them, and provide sensible service defaults.
- Readiness: select a navigation condition, selector wait, or delay based on the page. A network-idle condition is convenient for many pages but may wait indefinitely or capture too early on pages with unusual loading behavior.
- Viewport and full-page behavior: set dimensions explicitly when screenshots must be comparable. Use a full-page capture when the whole document matters; use an element screenshot when only a component is needed.
- Output format: choose the required image type and make sure downstream consumers can handle it. Include dimensions and format in job metadata or result records.
- Timeouts: keep navigation and whole-job limits finite. A page can load partially or hang; report the failure rather than letting a worker stay occupied without limit.
- State: decide whether every job starts with a fresh page/context or requires cookies, authentication, or a shared browser session. Do not share one user’s state with another job unintentionally.
How do I troubleshoot queued Puppeteer captures?
| Symptom | Likely cause | What to check or change |
|---|---|---|
| Jobs remain waiting | No worker is connected, the queue name or Redis configuration differs, or workers cannot reach the queue backend. | Check worker process health, queue name, connection settings, and backend connectivity. Confirm a worker is subscribed to the same queue used by the producer. |
| Jobs repeatedly fail after navigation | The page is slow, blocked, unreachable from the worker, or never reaches the selected readiness condition. | Capture the navigation error and timing details. Confirm the URL is reachable from the worker, then adjust the readiness condition or timeout to match the page rather than retrying indefinitely. |
| Screenshot is blank or incomplete | Capture occurs before meaningful content renders, the page is empty or blocked, or the selected capture area is wrong. | Inspect the page state at capture time; wait for a meaningful selector or appropriate readiness signal, and verify viewport and full-page or element settings. |
| Workers stop or memory pressure rises | Too many heavy pages run concurrently, browser processes leak, or cleanup is skipped on error. | Lower concurrency, ensure page/browser cleanup in a finally path, and observe resource use under the actual workload before raising limits. |
| Duplicate images or confusing status | Clients retried submission, a worker retried after partial completion, or results are not tied clearly to job IDs. | Adopt an idempotency policy and store job state and result references together; make result writes safe to repeat. |
Should I build a Puppeteer service or use a screenshot API?
Build when you need to own queue semantics, browser behavior, deployment, and result handling, and can operate those pieces. A hosted service can reduce browser and worker operations. The relevant distinction is whether you need a batch endpoint for straightforward captures or a stateful browser connection for interactive control.
Rank #4
| Approach | Best fit | Trade-off |
|---|---|---|
| ScreenshotNeo | Developers who want a screenshot API or MCP server without operating a browser queue. It removes known consent banners, newsletter popups, and chat widgets before capture; only clean shots are billed, and the lowest paid plan is $5. | Use its documented API or MCP workflow rather than designing your own worker and queue semantics. |
| Self-hosted Puppeteer with a queue | Teams that need to control job validation, retry policy, browser state, storage, and worker deployment. | Your team owns browser lifecycle, queue durability, security controls, resource limits, and operational recovery. |
| Screenshot API batch endpoint | Straightforward multi-URL jobs where an API can accept a batch and return a tracking ID. | Check that the provider’s batch behavior and capture controls fit your job model. The Screenshot API documentation describes a batch endpoint and tracking ID. |
| Hosted browser session | Cases needing stateful browser interaction controlled from Puppeteer or Playwright. | It provides a browser connection rather than necessarily replacing your own job orchestration. Capture says its browser sessions can provide a CDP connection URL. |
Compare durability and retries, request limits, capture controls, stateful interaction, infrastructure ownership, and integration effort. The cited product documentation does not establish current relative pricing, quotas, service guarantees, or comparative performance, so evaluate those directly before committing.
Or skip the browser setup
ScreenshotNeo provides a one-request capture API and an MCP server for AI agents. It removes cookie/consent banners, newsletter popups, and chat widgets before the screenshot; those cleanup steps can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits cost nothing, with response headers indicating the page verdict and billing status. An MCP server exposes screenshot tools to Claude, Cursor, and other MCP clients.
For a direct capture, create an API key and use the documented endpoint. See the ScreenshotNeo API documentation.
Best Value
curl -G "https://api.screenshotneo.com/v1/shot"
-d access_key=YOUR_API_KEY
--data-urlencode url=https://stripe.com
-o shot.webpThe free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 screenshots. Sign up for ScreenshotNeo’s free plan.
Frequently Asked Questions
Can I submit a large list of URLs at once?
BullMQ’s addBulk() accepts an array of jobs. Choose a batch size and worker capacity that fit your own limits; the API documentation does not prescribe a universal safe batch size.
Does adding a queue make captures reliable automatically?
No. Reliability depends on backend durability, retry and timeout policies, worker recovery, idempotency, and result storage.
Can I use a hosted browser and still write Puppeteer code?
Yes. Capture says its browser sessions can provide a CDP connection URL for Puppeteer or Playwright.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
