Build a crawler that uses ordinary HTTP for pages whose required content is already in the response, and launches headless Chrome only when a page needs JavaScript execution or browser interaction. With Node.js and Puppeteer, the core loop is: check scope and robots.txt, navigate to a page, wait for a target-specific readiness condition, extract the rendered DOM, and add eligible links to a bounded queue.
When a crawler needs headless Chrome
Headless Chrome runs without a visible user interface. Chrome’s current Headless mode shares the Chrome implementation with headful mode; since Chrome 132.0.6793.0, the old Headless implementation is available separately as chrome-headless-shell. See Chrome’s Headless documentation.
As an Amazon Associate I earn from qualifying purchases.
A browser is useful when the content or links you need are created by JavaScript or require browser interaction. If the server’s ordinary HTML response already contains the information, an HTTP client is usually a simpler retrieval path. If the site’s application already offers prerendering, consider using it rather than rendering every URL in a browser. Chrome discusses this distinction in its Puppeteer guidance.
Choose an automation tool
| Option | What it provides | When it fits |
|---|---|---|
| Puppeteer | JavaScript library for controlling Chrome or Firefox through DevTools Protocol or WebDriver BiDi. The official guide covers installation and a browser/page lifecycle. Puppeteer documentation. | A natural fit for a Node.js crawler centered on Chrome. Pin versions and make browser installation explicit in development and CI. |
| Playwright | Supports Chromium, a separate headless-shell download, newer Chromium Headless, and branded Chrome or Edge channels. Modes can behave differently. Playwright browser documentation. | Consider it when its broader browser tooling or cross-browser support fits the target sites. Specify which browser and Headless mode run in deployment. |
| Chrome command line | Chrome can be started with --headless; current Headless shares the Chrome browser implementation. Chrome Headless documentation. |
Useful for one-off automation or understanding the mode. A crawler with queues, extraction, and recovery generally benefits from an automation library. |
Choose based on runtime fit, browser binary management, mode fidelity, cross-browser needs, deployment footprint, and the automation APIs your target site requires. The sources cited here do not establish a comparable throughput or memory winner.
#1 Best Overall
- Includes Raspberry Pi 5 with 2.4Ghz 64-bit quad-core CPU (8GB RAM)
- Includes 128GB Micro SD Card pre-loaded with 64-bit Raspberry Pi OS, USB MicroSD Card Reader
- CanaKit Turbine Black Case for the Raspberry Pi 5
- CanaKit Low Noise Bearing System Fan
- Mega Heat Sink - Black Anodized
Set crawl boundaries and respect robots.txt
Before opening a browser, define the crawl’s purpose, seed hosts, and allowed URL scope. Do not collect authenticated or private material unless you are authorized to do so. Check the site’s terms and applicable rules as well as its crawler guidance.
robots.txt is crawler guidance, not access control or authorization. RFC 9309 requests that crawlers honor parseable rules; it does not grant permission to access a resource. See RFC 9309. Google also notes that disallowed URLs can still appear in search results and that robots.txt is not a way to secure information: Google’s robots.txt introduction.
- Fetch the host’s top-level
/robots.txtand identify your crawler with a descriptive user agent. - For a successfully fetched file, follow its parseable rules. RFC 9309 recommends following at least five consecutive redirects.
- If robots.txt is unavailable through a 4xx response, the RFC says a crawler may access resources. If it is unreachable because of a server or network error, such as a 5xx, assume complete disallow.
- Generally do not cache robots.txt for more than 24 hours, unless it is unreachable.
These are protocol details, not legal advice or a substitute for site-specific policies. Use password protection or another access control—not robots.txt—if your goal is to keep information private.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Build a small Puppeteer crawler
1. Install Puppeteer and its browser
In a new Node.js project, install Puppeteer:
npm install puppeteer
Puppeteer’s installation normally downloads a compatible browser. If your package manager or deployment environment blocks install scripts, the browser binary may be missing; explicitly allow the required install step or manage the browser binary as part of your deployment setup. Consult the Puppeteer installation guide.
2. Create a crawler file
Save this as crawler.mjs and run it with node crawler.mjs. It is a deliberately small example: it accepts one seed URL, limits crawling to that seed’s host, checks that host’s robots.txt, reuses one browser, and records extracted pages to standard output. The in-memory queue and visited set are suitable only for a small run; persist them for work that must survive a restart.
Rank #2
- Includes Raspberry Pi 4 4GB Model B with 1.5GHz 64-bit quad-core CPU (4GB RAM)
- Includes Pre-Loaded 32GB EVO+ Micro SD Card (Class 10), USB MicroSD Card Reader
- CanaKit Premium High-Gloss Raspberry Pi 4 Case with Integrated Fan Mount, CanaKit Low Noise Bearing System Fan
- CanaKit 3.5A USB-C Raspberry Pi 4 Power Supply (US Plug) with Noise Filter, Set of Heat Sinks, Display Cable - 6 foot (Supports up to 4K60p)
- CanaKit USB-C PiSwitch (On/Off Power Switch for Raspberry Pi 4)
import puppeteer from 'puppeteer';
const seed = new URL(process.argv[2] ?? 'https://example.com/');
const allowedHost = seed.host;
const userAgent = 'ExampleResearchCrawler/1.0 (+https://example.com/crawler-info)';
const maxPages = 20;
const timeoutMs = 30_000;
function normalize(raw, base) {
try {
const url = new URL(raw, base);
if (url.protocol !== 'http:' && url.protocol !== 'https:') return null;
url.hash = '';
if (url.host !== allowedHost) return null;
return url.href;
} catch {
return null;
}
}
async function readRobots(origin) {
const response = await fetch(new URL('/robots.txt', origin), {
headers: { 'user-agent': userAgent },
redirect: 'follow',
signal: AbortSignal.timeout(timeoutMs),
});
if (response.status >= 500) throw new Error(`robots.txt unreachable: ${response.status}`);
if (response.status >= 400) return '';
return response.text();
}
// This minimal matcher handles common User-agent, Allow, and Disallow lines.
// Use a maintained robots parser for production, especially for wildcard rules.
function allowedBySimpleRobots(text, path) {
let applies = false;
let best = null;
for (const rawLine of text.split(/r?n/)) {
const line = rawLine.split('#', 1)[0].trim();
const colon = line.indexOf(':');
if (colon < 0) continue;
const key = line.slice(0, colon).trim().toLowerCase();
const value = line.slice(colon + 1).trim();
if (key === 'user-agent') {
applies = value === '*' || userAgent.toLowerCase().includes(value.toLowerCase());
} else if (applies && (key === 'allow' || key === 'disallow') && value) {
if (path.startsWith(value) && (!best || value.length > best.path.length ||
(value.length === best.path.length && key === 'allow'))) {
best = { path: value, allow: key === 'allow' };
}
}
}
return best?.allow ?? true;
}
const robots = await readRobots(seed.origin);
const queue = [seed.href];
const seen = new Set();
const browser = await puppeteer.launch({ headless: true });
Rank #3
- 【What you Get】You will get 1*Pi 5 8GB Single Board,1*RasTech Case,1*Active Cooler,1*Screwdriver,1*Installation instructions,12-month free warranty, lifetime service, 24-hour prompt and friendly response.
- 【More Connectors】There are two USB 3.0 ports(5Gbps simultaneously) and two USB 2.0 ports, which triple total bandwidth ,support any combination of up to two cameras or displays. Peak SD card performance is doubled through support for the SDR104 high-speed mode. It provides a smooth desktop experience for you. Offer Gigabit Ethernet and a PCIe interface, along with dual-band Wi-Fi and Bluetooth 5.0/BLE wireless capability. The RasTech Pi 5 Kit use the new 27W 5.1V 5A USB-C power connector.
- 【 Support Dual 4Kp60 Display 】Each of the two microHDMI sockets can control a 4K display at 60 Hertz, now support HDR, offering super HD video for media streaming projects. RPi 5 is the first RPi model that comes with a PCI Express port (PCIe 2.0 x1 with 500 MB/s) to attach SSDs (requires separate M.2 HAT).
- 【 Excellent Chips And Applications】Pi 5 is a full-size Pi computer using silicon built in-house at Pi. The RP1 “southbridge” provides the bulk of the I/O capabilities for Pi 5. Pi 5 is more friendly and convenient in the development of Internet of Things, Web development, machine identification, automatic control and other electronic equipment applications and network.
- 【 Faster CPU, Better GPU 】 Pi 5 features a Broadcom BCM2712 64-bit quad-core Arm Cortex-A76 processor running at 2.4GHz, it delivers a 2–3× increase in CPU performance relative to RaspberryPi 4. The 800MHz VideoCore VII GPU is compatible to OpenGL ES 3.1 and Vulkan 1.2, substantial uplift in graphics performance. Pi 5 Offers lightning-fast CPU speed, a PCI Express interface, a Real Time Clock (RTC) and a power button and runs significantly cooler than Pi 4.
try {
while (queue.length && seen.size < maxPages) {
const url = queue.shift();
if (seen.has(url)) continue;
seen.add(url);
const parsed = new URL(url);
if (!allowedBySimpleRobots(robots, parsed.pathname + parsed.search)) {
console.log(JSON.stringify({ url, outcome: 'blocked-by-robots' }));
continue;
}
const page = await browser.newPage();
try {
await page.setUserAgent(userAgent);
const response = await page.goto(url, {
waitUntil: 'domcontentloaded',
timeout: timeoutMs,
});
// Replace this with a selector that signals readiness for your target site.
await page.waitForSelector('body', { timeout: 5_000 }).catch(() => {});
const result = await page.evaluate(() => ({
title: document.title,
text: document.body?.innerText ?? '',
links: Array.from(document.querySelectorAll('a[href]'), a => a.href),
}));
const finalUrl = page.url();
console.log(JSON.stringify({
url,
finalUrl,
fetchedAt: new Date().toISOString(),
status: response?.status() ?? null,
outcome: 'extracted',
title: result.title,
text: result.text,
}));
for (const link of result.links) {
const normalized = normalize(link, finalUrl);
if (normalized && !seen.has(normalized)) queue.push(normalized);
}
} catch (error) {
console.error(JSON.stringify({ url, outcome: 'error', error: String(error) }));
} finally {
await page.close();
}
}
} finally {
await browser.close();
}
Rank #4
- All-in-One Complete Kit: This SANOOV RPi 5 bundle comes with Raspberry Pi 5 4GB RAM single board, active cooler, durable ABS case and screwdriver. No extra parts needed, ready to use right out of the box for beginners and hobbyists
- Powerful Single Board Computer: Equipped with 4GB RAM and high-performance processor, delivers fast running speed for 4K playback, AI projects, programming and daily computing tasks. SANOOV for raspberry pi 5 4GB is equipped with broadcom 64 quad-core Arm Cortex A76 processor with gigabit ethernet and upgraded with IEEE 802.11ac Wi-Fi, Bluetooth 5.0 dual-band 2.4Ghz and 5Ghz and Power Over Ethernet (POE). Upgrading delivers 2-3 x speed vs Pi 4, redefining the experience
- Efficient Active Cooler: Effectively lowers operating temperature and prevents performance throttling. Runs quietly even under long-time heavy load, ensures stable operation all day long. SANOOV RPi 5 4GB kit offer an active cooler, which combines an aluminium heatsink with a high-performance PWM fan. Active cooler is fully compatible with the Pi OS, which can effectively reduce the temperature of RPi5 and ensure its good performance during long-term high load operation
- Sturdy ABS Protective Case: Well-fitted for Raspberry Pi 5 board, can be secured with 4 screws to effectively protect the Pi 5 motherboard from damage, reserves full access to all ports and buttons. SANOOV uses ABS material to produce the case, which has a softer texture and feel. Meanwhile, SANOOV case adopts a layered design for easy disassembly and installation. (Tip: The Case cannot install M.2 HAT Add on Board and Solid State Drive!)
- Wide Application & Full Compatibility: Seamlessly compatible with official OS and mainstream peripheral accessories for Raspberry Pi 5. Whether you are a beginner, student, electronics hobbyist or professional developer, this all-in-one kit meets your diverse needs. It excels in IoT projects, robotics design, retro gaming devices, home media servers and other DIY creations. Backed by a large global community, you can easily find guides, technical support and shared projects online
The example’s robots matcher is intentionally limited; it is not a complete implementation of RFC 9309. Use a maintained parser for production, including correct handling of groups, wildcards, and matching semantics. The example treats a 4xx robots response as unavailable and stops on a 5xx/network error, but a production implementation should explicitly classify fetch failures and apply the protocol rules.
3. Tune readiness to the page
domcontentloaded is a navigation milestone, not proof that a JavaScript application has finished rendering. Prefer a meaningful readiness condition such as a target selector or known page state, with a bounded timeout. Use a fixed delay only when you have a reason and a limit. A generic “network idle” wait can be misleading on pages that keep connections open or load content later; choose based on the content you need. Chrome’s example demonstrates navigating and reading serialized page content, but does not make one wait condition universal: Chrome Puppeteer guidance.
Scale cautiously and make results auditable
Bound browser work
The example reuses one browser and opens one page at a time. For more throughput, use a bounded pool of pages or browser workers rather than launching an unbounded number of processes. Close pages after each task and close browsers when the run ends. There is no universal safe request rate established here: pace requests per host according to its policy and observed response behavior.
Retry and persist deliberately
- Set navigation and operation timeouts. Retry transient failures only a limited number of times with backoff; stop retrying persistent errors.
- Keep queue and visited state outside the browser process if a crawl must recover after interruption.
- Record original URL, final URL, fetch time, response status when available, and extraction outcome.
- Observe queue depth, successes, errors, render time, and duplicate rate to understand coverage and resource use.
These are engineering practices rather than performance guarantees. The cited sources do not prescribe a distributed crawler architecture or a universal request pace.
Free tools Windows power users keep installed
One-click scans. No signup required.
Filter resources only after validation
Request interception can avoid loading resources a task does not need; Chrome’s example shows allowing documents, scripts, XHR, and fetch while aborting other types. But blocking images, stylesheets, or other resources can break sites whose rendering depends on them. Compare extracted output with and without filtering before adopting a rule. See Chrome’s Puppeteer article.
Best Value
- Not including the Raspberry Pi 5 (8GB), the Crowpi advanced version comes with the Raspberry Pi 5
- ELECROW Black Case for the Raspberry Pi 5, CrowPi is equipped with a 9-inch HD touchscreen along with a camera; All the regular components used in DIY electronics are packed into the CrowPi development board, such as LCD, LED matrix, buzzer, light sensor, PIR sensor, ultrasonic sensor, IR sensor, etc
- Raspberry Pi Sensors: The Crowpi raspberry pi 5 programming kit is jam-packed with lots of buttons such as 19 different sensors in a tidy easy to use package; You don't have to wait and wire things
- Build Quality: Solid ABS shell and well made components in one place make it strong and convenient to travel
- Programming Lessons: This raspberry pi 5 learning kit ships with step by step instructions and provides 21 lessons to take you through identifying components reading code and running it in the terminal
Troubleshoot common failures
| Symptom | Likely cause | What to check or change |
|---|---|---|
| Puppeteer launches but cannot find Chrome | The browser download or install script did not run, or deployment does not include the browser binary. | Check the Puppeteer install output and deployment image; install the compatible browser explicitly. See the installation guide. |
| Page text is empty or incomplete | The page has not rendered the target content at the point extraction runs, or the content requires interaction. | Wait for a target-specific selector or state, and perform only authorized interactions needed for the page. |
| Navigation times out | The page is slow, a resource never settles, or the chosen readiness condition is too broad. | Use a bounded timeout and a more relevant readiness signal; inspect whether the target content became available before the timeout. |
| Links lead outside the intended crawl | Relative URLs, fragments, query strings, or external hosts are not being normalized and filtered consistently. | Resolve links against the final page URL, normalize according to your crawl policy, and reject unsupported schemes and disallowed hosts before queueing. |
| A site breaks after request blocking | Rendering depends on a blocked resource. | Restore the resource types and test extraction again before narrowing filters. |
| Robots rules seem to conflict | A simplified parser may mishandle groups, wildcards, or matching precedence. | Use a maintained robots parser and apply the RFC’s retrieval and caching rules; do not treat the file as authorization. |
Or skip the browser setup
If you need screenshots rather than a custom DOM-crawling pipeline, ScreenshotNeo is a website screenshot API and MCP server for developers. One GET request returns a PNG, JPEG, WebP, or PDF. The example below saves a WebP screenshot of a page:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
See the ScreenshotNeo API documentation for request options. It accepts cookie or consent banners as a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and responses identify the page verdict and billing status in headers. Its MCP server gives AI agents tools for taking screenshots, getting page information, and capturing PDFs. The Free plan includes 1,000 shots per month without a card; paid plans start at $5 for 3,000 shots.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchSign up for ScreenshotNeo’s free plan: 1,000 screenshots a month, no card required.
Frequently Asked Questions
Does robots.txt give permission to crawl a page?
No. It is crawler guidance, not authorization or access control. Check the site’s terms and applicable rules.
Can I use Playwright instead of Puppeteer?
Yes. Playwright is a credible alternative; choose and configure the browser and Headless mode explicitly because its documented modes can differ.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →
