To bulk screenshot websites without repeating work, preserve each original URL, create a separate conservative deduplication key, capture each unique key with a browser, and record the result for every input row. Do not treat similar-looking URLs as duplicates by default: query strings and, in some sites, URL fragments can select different content.
Build a safe bulk-capture workflow
- Collect and validate the URL list. Keep the original URL unchanged for logs and output naming. Reject malformed or unsupported inputs explicitly rather than silently rewriting them.
- Generate a comparison key. Normalize only URI equivalences you have chosen to treat as identical. Keep the original alongside the key.
- Capture one URL per unique key. Use fixed browser settings, controlled concurrency, and explicit navigation and screenshot timeouts.
- Write a manifest. For every input, record its original URL, comparison key, output filename, final URL if navigation succeeded, status, and any error. Mark skipped duplicates and point them to the first matching capture.
This separates the question “are these strings equivalent for my job?” from “did the page render correctly?” It also lets a later run retry only failures without losing the original list.
As an Amazon Associate I earn from qualifying purchases.
How to skip duplicate URLs safely
Keep the original and deduplication key separate
Use the original URL as the record of what was submitted. Derive a separate key for comparison. RFC 3986 treats the scheme and host as case-insensitive in generic URI syntax, but other components are generally case-sensitive unless a scheme specifies otherwise. It also describes normalization such as uppercasing hexadecimal digits in percent-encoded octets, decoding percent-encoded unreserved characters, and removing dot segments where applicable. See RFC 3986.
Do not decode reserved characters or casually reorder a query string. Reserved characters can delimit URI components, and queries often contain identifiers or state. Sorting parameters or removing values that look like tracking tags is an application-specific rule; do it only when you know the target treats the resulting URLs as equivalent.
#1 Best Overall
- CRISP CLARITY: This 23.8″ Philips V line monitor delivers crisp Full HD 1920x1080 visuals. Enjoy movies, shows and videos with remarkable detail
- INCREDIBLE CONTRAST: The VA panel produces brighter whites and deeper blacks. You get true-to-life images and more gradients with 16.7 million colors
- THE PERFECT VIEW: The 178/178 degree extra wide viewing angle prevents the shifting of colors when viewed from an offset angle, so you always get consistent colors
- WORK SEAMLESSLY: This sleek monitor is virtually bezel-free on three sides, so the screen looks even bigger for the viewer. This minimalistic design also allows for seamless multi-monitor setups that enhance your workflow and boost productivity
- A BETTER READING EXPERIENCE: For busy office workers, EasyRead mode provides a more paper-like experience for when viewing lengthy documents
Decide what a fragment means for your target
In the generic URI model, the fragment identifies a secondary resource or view of a representation and is not part of the retrieval action sent to the server. For ordinary server-rendered pages, removing the fragment from the key may prevent repeated downloads of the same document. But browser-based applications can use fragments to choose a view or state. If the screenshot should capture each fragment-specific view, retain fragments in the comparison key.
Do not confuse duplicate strings with duplicate content
Exact duplicate strings are straightforward to skip. Near-duplicate content is a separate problem: protocol, regional, device, sorting, or filtering variants may show similar pages but still be intentionally distinct. Google’s canonicalization guidance explains how a site can choose a representative URL among duplicate or similar pages; that content-level decision cannot safely be inferred from URL text alone. See Google’s canonicalization documentation.
Capture a URL list locally with Playwright
Playwright’s documented browser workflow is to launch a browser, open a page, navigate to a URL, then save a screenshot. Its screenshot API supports full-page capture and returning an image buffer. The example below uses an exact-string deduplication key to avoid making unsafe assumptions about URL equivalence. Replace that function only if your application has a documented normalization policy.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteRank #2
- CRISP CLARITY: This 22 inch class (21.5″ viewable) Philips V line monitor delivers crisp Full HD 1920x1080 visuals. Enjoy movies, shows and videos with remarkable detail
- 100HZ FAST REFRESH RATE: 100Hz brings your favorite movies and video games to life. Stream, binge, and play effortlessly
- SMOOTH ACTION WITH ADAPTIVE-SYNC: Adaptive-Sync technology ensures fluid action sequences and rapid response time. Every frame will be rendered smoothly with crystal clarity and without stutter
- INCREDIBLE CONTRAST: The VA panel produces brighter whites and deeper blacks. You get true-to-life images and more gradients with 16.7 million colors
- THE PERFECT VIEW: The 178/178 degree extra wide viewing angle prevents the shifting of colors when viewed from an offset angle, so you always get consistent colors
Install the prerequisites
With Node.js installed, create a project and install Playwright:
npm init -y
npm install playwright
npx playwright install chromiumSave the following as bulk-screenshots.mjs. Put one URL per line in urls.txt.
import { chromium } from 'playwright';
import { createHash } from 'node:crypto';
import { mkdir, readFile, writeFile } from 'node:fs/promises';
const inputPath = process.argv[2] ?? 'urls.txt';
const outputDir = process.argv[3] ?? 'screenshots';
const concurrency = Number(process.env.CONCURRENCY ?? 3);
const navigationTimeoutMs = Number(process.env.NAVIGATION_TIMEOUT_MS ?? 30000);
const fullPage = process.env.FULL_PAGE !== 'false';
if (!Number.isInteger(concurrency) || concurrency < 1) {
throw new Error('CONCURRENCY must be a positive integer');
}
const lines = (await readFile(inputPath, 'utf8')).split(/r?n/);
const urls = lines.map((line, index) => ({ line: index + 1, original: line.trim() }))
.filter(item => item.original && !item.original.startsWith('#'));
function validateUrl(value) {
try {
const parsed = new URL(value);
if (!['http:', 'https:'].includes(parsed.protocol)) return 'Only http and https URLs are supported';
return null;
} catch {
return 'Invalid URL';
}
}
// Conservative by design: only identical input strings are duplicates.
function comparisonKey(original) {
return original;
}
function filenameFor(original) {
return createHash('sha256').update(original).digest('hex').slice(0, 20) + '.png';
}
await mkdir(outputDir, { recursive: true });
const seen = new Map();
const records = urls.map(item => {
const error = validateUrl(item.original);
const key = comparisonKey(item.original);
const prior = seen.get(key);
if (!error && prior) {
return { ...item, key, status: 'duplicate', duplicateOfLine: prior.line, file: prior.file };
}
const record = {
...item,
key,
file: error ? null : filenameFor(item.original),
status: error ? 'invalid' : 'queued',
error: error ?? null
};
if (!error) seen.set(key, { line: item.line, file: record.file });
return record;
});
const browser = await chromium.launch({ headless: true });
let next = 0;
async function worker() {
while (true) {
const index = next++;
if (index >= records.length) return;
const record = records[index];
if (record.status !== 'queued') continue;
const page = await browser.newPage({ viewport: { width: 1440, height: 1000 }, deviceScaleFactor: 1 });
try {
const response = await page.goto(record.original, {
waitUntil: 'networkidle',
timeout: navigationTimeoutMs
});
record.finalUrl = page.url();
record.httpStatus = response?.status() ?? null;
await page.screenshot({ path: `${outputDir}/${record.file}`, fullPage });
record.status = 'captured';
} catch (error) {
record.status = 'failed';
record.error = String(error?.message ?? error);
} finally {
await page.close();
}
}
}
try {
await Promise.all(Array.from({ length: Math.min(concurrency, records.length) }, () => worker()));
} finally {
await browser.close();
}
await writeFile(`${outputDir}/manifest.json`, JSON.stringify({
generatedAt: new Date().toISOString(),
settings: { browser: 'Chromium via Playwright', viewport: { width: 1440, height: 1000 }, deviceScaleFactor: 1, fullPage, concurrency, navigationTimeoutMs },
records
}, null, 2));
console.log(`Processed ${records.length} inputs. See ${outputDir}/manifest.json`);Run it with node bulk-screenshots.mjs urls.txt screenshots. Set CONCURRENCY to a small positive number to limit simultaneous pages; set NAVIGATION_TIMEOUT_MS to adjust the navigation timeout, and set FULL_PAGE=false for viewport-only screenshots. The manifest distinguishes captured pages, invalid URLs, failures, and duplicate inputs.
Rank #3
- Clear visuals. Fluid motion: A 144Hz refresh rate and 1ms MPRT deliver smooth, tear‑free motion across work, gaming, and streaming for clearer, more fluid viewing.
- Eye comfort: TÜV Rheinland 3‑star* certification reduces harmful blue light while preserving stunning color quality without compromise. *TÜV Rheinland 3-star eye comfort certification.
- Wide viewing angle: Get consistent views across a wide 178° /178° viewing angle.
- In-Plane Switching (IPS): See excellent color accuracy and consistency across wide viewing angles with In-plane Switching (IPS) technology.
- Ultra-thin bezels: Maximize your viewing experience with thin bezels.
Choose the wait condition deliberately
The example waits for networkidle, which may suit pages that settle after requests finish, but some sites keep connections open or continue background activity. If this makes captures time out, choose a less strict navigation condition such as domcontentloaded, then wait for a site-specific selector or a deliberate delay before taking the screenshot. A selector is often better when the useful content appears after client-side rendering.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Make repeated runs comparable
A fixed viewport, browser, device scale factor, and headless setting reduce variation, but do not guarantee pixel-identical output. Playwright notes that rendering can vary with host operating system, browser version, settings, hardware, power source, and headless mode. Record the environment with the output and keep it stable if screenshots will be compared over time. See Playwright’s visual comparison guidance.
When a managed bulk API makes sense
Local Playwright gives you direct control over browser settings, naming, storage, and error handling, while requiring you to maintain the browser environment, concurrency, retries, and storage. A managed service can reduce that maintenance and may accept URL lists or sitemaps, but verify its current batch limits, output handling, security, failure reporting, terms, and price before choosing it.
Rank #4
- CURVED FOR ENHANCED ENGAGEMENT: An immersive viewing experience with a curved monitor that wraps more closely around your field of vision; It creates a wider view, enhancing depth perception and minimizing peripheral distraction
- SMOOTH PERFORMANCE FOR SEAMLESS CONTENT: Stay in the action when playing games, watching videos, or working on creative projects; The 100Hz refresh rate reduces lag and motion blur so you don't miss a thing in fast-paced moments¹
- MORE GAMING POWER: Gain the edge with optimizable game settings; Color and image contrast can be adjusted to see scenes more vividly and spot enemies hiding in the dark; Game Mode adjusts any game to fill the screen so you can view every detail²
- KEEP IT EASY ON THE EYES: Care for your eyes and stay comfortable, even during long sessions; Advanced eye comfort technology certified by TÜV reduces eye strain by minimizing blue light and reducing irritating screen flicker²
- INCREASED VERSATILITY: Connect to more; Plug devices straight into your monitor for increased flexibility, making your computing environment even more convenient
The AddScreenshots Swagger UI documents asynchronous bulk endpoints for multiple URLs, sitemaps or URL-list files, and whole domains, with background processing and cloud storage options such as S3 or Azure Blob Storage. This documentation describes those capabilities; it is not an independent performance test or confirmation of current limits, prices, or service terms. See AddScreenshots Swagger UI.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Or skip the browser setup
ScreenshotNeo is a website screenshot API and MCP server. One GET request can return a PNG, JPEG, WebP, or PDF; its bulk capture option accepts up to 100 URLs per call. Cookie and consent banners, newsletter popups, and chat widgets can be removed before capture, and each cleanup step can be disabled. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed; response headers identify the page verdict and billing status. AI agents can use its MCP server tools, including take_screenshot, get_page_info, and capture_pdf. Plans include 1,000 screenshots a month free with no card and paid plans starting at $5 for 3,000.
For one URL, adapt the URL parameter to your target. See the ScreenshotNeo API documentation for request options and response details.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpFor a recurring URL-list job, submit URLs in supported batches, retain your input list, and save a per-URL manifest so duplicate handling and failures remain auditable. Learn more at ScreenshotNeo, or sign up for 1,000 free screenshots a month with no card.
Best Value
- 【INTEGRATED SPEAKERS】Whether you're at work or in the midst of an intense gaming session, our built-in speakers provide rich and seamless audio, all while keeping your desk clutter-free.
- 【EASY ON THE EYES】 Protect your eyes and enhance your comfort with Blue-Light Shift technology. This feature reduces harmful blue light emissions from your screen, helping to alleviate eye strain during long hours of use and promoting healthier viewing habits.
- 【WIDEN YOUR PERSPECTIVE】Our sleek minimal bezel design ensures undivided attention. The nearly bezel-free display seamlessly connects in a dual monitor arrangement, delivering an unobstructed view that lets you focus on more at once, completely distraction-free.
Troubleshooting bulk captures
- A URL fails validation: Check whitespace, scheme, and spelling. This script accepts only absolute
httpandhttpsURLs. - Navigation times out: The site may be slow, blocked, or never reach the selected wait condition. Increase the timeout if appropriate or use a less strict wait condition followed by a targeted selector wait.
- The screenshot misses content: The page may render after the initial load. Wait for the content selector or a bounded delay before capture; do not assume that navigation completion means all lazy or client-rendered content is ready.
- Repeated captures differ: Check viewport, browser version, operating system, device scale, headless mode, and other environment settings. Record these with the manifest.
- Runs overload the target or machine: Lower
CONCURRENCY. An unlimited number of browser pages can consume resources and generate unwanted request load. - Two URLs look alike but should both be captured: Use a key policy that preserves meaningful query parameters or fragments. Do not sort or remove query values unless the target application treats those URLs identically.
- A retry overwrites an earlier file: The example uses a deterministic filename derived from the original URL. Copy or version prior output directories if you need to retain multiple runs.
FAQ
Should I remove URL fragments when deduplicating?
Only if the target uses fragments as ordinary client-side state that is irrelevant to the capture. Keep them when each fragment selects a view you need to screenshot.
Does a successful screenshot prove the page is the intended canonical page?
No. A capture records what the browser rendered after navigation. Canonical-page selection is a site-level content decision, not something a URL-string deduplicator can determine reliably.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
