Free tools Windows power users keep installed
One-click scans. No signup required.
Use a headless browser when the data appears only after JavaScript runs or when scraping requires real browser actions. Playwright, Puppeteer, and Selenium can open pages without displaying a window, execute scripts, click controls, wait for network activity, and read the rendered DOM. If a normal HTTP request already returns the data you need, use that simpler approach instead: browser automation adds browser binaries, memory, startup time, failure modes, and deployment work.
This guide explains the decision, compares the main frameworks and managed services, and gives production-minded examples. It does not make scraping lawful by itself; check the target site’s terms, robots guidance, privacy obligations, and applicable law for your use case.
What a headless browser actually does
A headless browser is a browser engine controlled by code rather than a visible desktop window. It performs the same broad sequence as a normal visit: resolve the URL, download resources, build the DOM, run JavaScript, apply styles, create cookies and storage, and expose the resulting page to automation code. Your script can then inspect text or attributes and interact with the page.
- Navigation: open URLs, follow links, submit forms, and manage redirects.
- Rendering: execute client-side JavaScript and wait for a usable state.
- Interaction: click, type, scroll, select options, and upload files.
- Observation: read the rendered DOM, network responses, console messages, cookies, and screenshots.
Headless does not mean invisible to a website. Sites can still detect automation, require authentication, issue bot challenges, or restrict access. A headless session also does not bypass authorization or make collection permissible.
#1 Best Overall
Decide whether you need a browser
Start with a direct HTTP request
Fetch the page or its underlying JSON endpoint with an HTTP client first. Inspect the response body, scripts, and network calls in a normal browser’s developer tools. If the required records are already present in HTML or available from an authorized endpoint, an HTTP scraper is usually easier to run and scale. ProxiesAPI Guides characterizes headless browsers as slower, heavier, and harder to scale than plain HTTP scraping; that is vendor guidance, not an independently measured benchmark.
Use a browser when the task is browser-specific
- The initial HTML is an application shell and records arrive through JavaScript requests.
- You must click “load more,” paginate through a virtualized list, or open a menu before data exists.
- Content depends on cookies, local storage, a logged-in session, or a selected location.
- You need screenshots, PDFs, layout information, or an element’s visible state.
- The site’s documented workflow requires form submission or another user interaction.
Measure the real cost
Each worker needs a browser binary and resources for pages, tabs, images, fonts, and JavaScript. Startup and navigation failures are more varied than an HTTP 500: a selector can change, a page can remain in a loading state, or a challenge can replace the content. Limit concurrency, reuse browser processes where appropriate, set explicit timeouts, and record why each job failed.
Playwright, Puppeteer, and Selenium compared
| Option | Documented strengths | Best-fit questions |
|---|---|---|
| Playwright | Projects for Chromium, Firefox, and WebKit. Its documentation distinguishes Chromium headless shell from the newer Chromium headless mode. | Do you need cross-browser checks or a particular Chromium headless mode? Test the exact browser build and mode against the target. |
| Puppeteer | JavaScript-centered automation for Chrome and Firefox, using Chrome DevTools Protocol and WebDriver BiDi. It supports page interaction and screenshots. | Is your application JavaScript-based, and is Chrome/Firefox plus Puppeteer’s protocol scope sufficient? |
| Selenium | The Puppeteer FAQ describes broader language bindings and orchestration tooling such as Selenium Grid. | Do you need an established multi-language ecosystem or distributed Grid-style orchestration? |
Official documentation establishes capability differences, not a universal speed or reliability winner. Choose by browser coverage, programming language, interaction complexity, compatibility requirements, deployment model, and the team’s ability to operate browsers.
Playwright example: render and extract a JavaScript page
The following Python script opens a page, waits for a selector, extracts links, and closes the browser. Install Playwright and its browser binaries first:
Recommended Free Tools
pip install playwright
playwright install chromiumfrom playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeoutError
URL = "https://example.com/catalog"
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page(viewport={"width": 1440, "height": 900})
page.set_default_timeout(15_000)
try:
page.goto(URL, wait_until="domcontentloaded", timeout=45_000)
page.locator(".product-card").first.wait_for(state="visible")
products = page.locator(".product-card").evaluate_all("""
cards => cards.map(card => ({
name: card.querySelector('.name')?.textContent?.trim() ?? null,
price: card.querySelector('.price')?.textContent?.trim() ?? null,
href: card.querySelector('a')?.href ?? null
}))
""")
print(products)
except PlaywrightTimeoutError:
page.screenshot(path="debug-timeout.png", full_page=True)
raise
finally:
browser.close()Replace selectors with stable attributes from the target site. Prefer waiting for a meaningful element or response rather than sleeping for an arbitrary number of seconds. If a page uses infinite scroll, scroll in bounded increments and stop when the item count no longer increases or an explicit end marker appears.
Equivalent Node.js patterns
Playwright
import { chromium } from 'playwright';
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
try {
await page.goto('https://example.com/catalog', { waitUntil: 'domcontentloaded', timeout: 45000 });
await page.locator('.product-card').first().waitFor({ state: 'visible', timeout: 15000 });
const products = await page.locator('.product-card').evaluateAll(cards => cards.map(card => ({
name: card.querySelector('.name')?.textContent?.trim() ?? null,
price: card.querySelector('.price')?.textContent?.trim() ?? null,
href: card.querySelector('a')?.href ?? null
})));
console.log(products);
} finally {
await browser.close();
}Puppeteer
import puppeteer from 'puppeteer';
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
try {
await page.goto('https://example.com/catalog', { waitUntil: 'domcontentloaded', timeout: 45000 });
await page.waitForSelector('.product-card', { visible: true, timeout: 15000 });
const products = await page.$$eval('.product-card', cards => cards.map(card => ({
name: card.querySelector('.name')?.textContent?.trim() ?? null,
price: card.querySelector('.price')?.textContent?.trim() ?? null,
href: card.querySelector('a')?.href ?? null
})));
console.log(products);
} finally {
await browser.close();
}Production controls that prevent fragile scrapers
Wait for state, not time
Use a selector, a response predicate, or a framework’s network-idle condition when it represents readiness. Network idle can be misleading on pages with analytics or long polling, so combine it with a domain-specific selector.
Make selectors resilient
Prefer semantic roles, labels, stable data attributes, and URL patterns. Avoid generated class names and deeply nested CSS paths. Keep selectors in configuration so a markup change does not require rewriting the crawler.
Control sessions and data
Create an isolated context for each account or tenant. Store cookies securely, never log credentials, and clear contexts when a job ends. Set locale, timezone, viewport, and user agent deliberately because they can change both content and layout.
Throttle and retry carefully
Bound concurrency per host, add exponential backoff for transient failures, and distinguish navigation timeout, selector timeout, HTTP error, empty result, and bot challenge. Retrying an authorization failure or a challenge repeatedly can worsen the problem. Cache immutable pages and checkpoint pagination so a worker can resume.
Capture evidence
On failure, retain a timestamp, URL, status, final URL, console errors, a sanitized screenshot, and a short HTML excerpt. Redact tokens and personal data. This makes selector regressions and consent-page substitutions diagnosable.
Local browsers versus managed browser services
Running Playwright, Puppeteer, or Selenium yourself gives control over versions, extensions, network routing, credentials, and data locality. You also own browser downloads, sandbox settings, container images, scaling, crash recovery, observability, and security patching.
Managed services move some of that work to a provider. Browserless documents managed browser infrastructure, Puppeteer and Playwright connections, and APIs for scraping and other browser tasks. Cloudflare Browser Run documents a headless Chrome service with Quick Actions and scripted sessions through Playwright, Puppeteer, CDP, or Stagehand; its documentation page was last updated August 11, 2026. Before committing, verify current limits, regions, retention, authentication, concurrency, browser versions, and pricing for your workload.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Or skip the browser setup
For screenshots and PDFs rather than custom extraction workflows, ScreenshotNeo is the first service to try: it produces clean shots, bills only clean shots, and its paid entry plan is $5.
One GET request is enough:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo documentation for parameters and response details. The same request in Python:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)And Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);ScreenshotNeo accepts cookie and consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be switched off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing result. Its MCP server supplies take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.
It also supports full-page captures with lazy images loaded, CSS-selector element captures, dark mode, 12 device presets plus custom viewports, retina scale, PDF paper sizes and ranges, custom CSS and JavaScript, clicks, waits, blocking rules, headers, cookies, user agents, Authorization, timezone and geolocation, transparent backgrounds, resizing, chosen cache TTLs, signed image links, asynchronous jobs with signed webhooks, bulk capture of up to 100 URLs per call, a usage API, and an OpenAPI specification. Parameter names used by other screenshot APIs also work.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →| Plan | Included shots | Price |
|---|---|---|
| Free | 1,000 per month | $0, no card |
| Starter | 3,000 | $5 |
| Growth | 15,000 | $15 |
| Pro | 60,000 | $39 |
| Scale | 250,000 | $99 |
| Business | 1,000,000 | $249 |
Yearly billing provides two months free, and every feature is available on every plan. Create a free ScreenshotNeo account with 1,000 screenshots a month and no card.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshooting common failures
“Browser executable not found”
Install the framework’s browser binaries in the same environment that runs the job (for example, playwright install chromium). In containers, bake them into the image rather than downloading on every request.
Navigation times out
Check DNS, proxy and firewall rules, then inspect the final URL and response status. Raise the timeout only after confirming the page is legitimately slow. Use domcontentloaded plus a readiness selector instead of waiting forever for every resource.
The selector never appears
Save a screenshot and HTML from the failing run. You may be on a consent page, login screen, alternate locale, bot challenge, or changed markup. Verify the frame: content inside an iframe requires selecting that frame before querying.
Results are empty but the page looks correct
Inspect network responses and the rendered DOM, not just the original response body. Virtualized lists may keep only visible rows in the DOM; scroll and extract incrementally, or use an authorized data endpoint discovered from the page.
Best Value
Works locally, fails in a container
Compare browser version, OS libraries, sandbox permissions, fonts, viewport, timezone, and outbound network access. Use a supported base image, run as a least-privileged user with the required sandbox configuration, and log the exact versions.
Choosing a practical architecture
- Classify the source: static HTML or API means HTTP; rendered or interactive means browser.
- Select the framework: Playwright for documented Chromium/Firefox/WebKit projects, Puppeteer for a JavaScript-centered Chrome/Firefox workflow, or Selenium when language bindings and Grid-style orchestration matter.
- Prototype one representative page: record readiness signals, authentication steps, pagination behavior, and failure screenshots.
- Set operating limits: timeouts, concurrency, memory ceilings, retry rules, retention, and per-host rate limits.
- Validate continuously: monitor empty-result rates and selector failures, and rerun against known fixtures after browser or site changes.
No framework is reliable by name alone. Reliability comes from matching the browser mode and interaction model to the target, isolating sessions, observing failures, and having a recovery path when the site changes.
Frequently Asked Questions
Can a headless browser scrape any website?
No. A browser can render and interact only as far as the site permits. Authentication, bot checks, technical barriers, terms, privacy rules, and local law still apply.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Is headless scraping always slower than HTTP scraping?
It generally carries more browser and rendering overhead, but the supplied vendor comparison provides no independent benchmark. Measure your own pages and workload.
Which framework should a beginner learn first?
Choose Playwright for documented multi-browser projects, Puppeteer for a JavaScript-centered Chrome/Firefox workflow, or Selenium when your team needs its broader language and Grid ecosystem.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
