Recommended Free Tools
To loop through matching elements in Puppeteer, wait for the content you need and call page.$$eval(). It sends every matching element to one browser-context callback, where you map each node to plain strings or objects that Node.js can use:
await page.waitForSelector('.product-card', { visible: true, timeout: 15000 });
const products = await page.$$eval('.product-card', cards =>
cards.map(card => ({
name: card.querySelector('.name')?.textContent?.trim() ?? '',
price: card.querySelector('.price')?.textContent?.trim() ?? '',
href: card.querySelector('a')?.href ?? null
}))
);Use page.$$() instead when you need element handles for interaction or per-item error handling. Use page.$eval() only when exactly one match is expected; it throws if there is no match.
Set up a Puppeteer scraper
Install Puppeteer in a Node.js project. The package downloads a compatible Chromium build unless you configure it to use another browser.
mkdir puppeteer-loop
cd puppeteer-loop
npm init -y
npm install puppeteerCreate scrape.js and run it with node scrape.js. The examples below use a public URL placeholder; replace it with a page you are allowed to access.
#1 Best Overall
Loop through all matching elements with $$eval
$$eval(selector, pageFunction) finds every match and passes the resulting array to pageFunction. The callback runs inside the browser, so it can read DOM properties such as textContent, attributes and links directly. Return serializable values rather than live DOM nodes.
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({ headless: true });
try {
const page = await browser.newPage();
await page.goto('https://example.com/products', {
waitUntil: 'domcontentloaded',
timeout: 30_000
});
await page.waitForSelector('.product-card', {
visible: true,
timeout: 15_000
});
const products = await page.$$eval('.product-card', cards =>
cards.map(card => ({
name: card.querySelector('.name')?.textContent?.trim() ?? '',
price: card.querySelector('.price')?.textContent?.trim() ?? '',
href: card.querySelector('a')?.href ?? null,
sku: card.getAttribute('data-sku') ?? null
}))
);
console.log(JSON.stringify(products, null, 2));
} finally {
await browser.close();
}
})();If the callback itself is asynchronous, Puppeteer waits for its promise:
const records = await page.$$eval('.row', async rows => {
return rows.map(row => ({
id: row.getAttribute('data-id'),
label: row.textContent?.trim() ?? ''
}));
});Read the right value
- Use
textContent?.trim() ?? ''for text while tolerating a missing child. - Use
element.hreffor an absolute link resolved by the browser. - Use
getAttribute()for data attributes and values that may legitimately be absent. - Convert numbers explicitly in Node.js after extraction; visible currency symbols and localized separators need deliberate parsing.
Wait for dynamic elements before scraping
Navigation finishing does not guarantee that JavaScript-rendered cards or rows exist. Synchronize on a selector that represents the data, not merely on a fixed sleep.
await page.goto('https://example.com/results', {
waitUntil: 'domcontentloaded',
timeout: 30_000
});
await page.waitForSelector('.results', {
visible: true,
timeout: 15_000
});
const rows = await page.$$eval('.results tr', trs =>
trs.map(tr => [...tr.querySelectorAll('td')]
.map(td => td.textContent?.trim() ?? ''))
);waitForSelector works across navigations and can be configured with visible, hidden and a bounded timeout. If the selector never appears, it throws, allowing your script to report a failed extraction instead of silently saving an empty file. A selector can also represent a loading state disappearing:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
await page.waitForSelector('.spinner', { hidden: true, timeout: 20_000 });
await page.waitForSelector('.result-card', { visible: true, timeout: 10_000 });Prefer semantic selectors such as data-testid, accessible roles exposed through stable attributes, or meaningful class names. Positional selectors and generated CSS class names break when a site redesigns its markup.
Choose between $$eval, $$ and $eval
| Method | Callback or handle location | Best use | No-match behavior |
|---|---|---|---|
$$eval |
One callback in the browser page | Bulk extraction that can be mapped to plain data | Returns an empty array |
$$ |
Node.js receives an array of ElementHandles | Interactions, sequencing, per-element failures or handle-based work | Resolves to [] |
$eval |
One browser callback for the first match | A required single value such as one heading | Throws when no element is found |
Use $$ for controlled iteration
Element handles let Node.js click, inspect or process one item at a time. Dispose of each handle when finished, especially for large result sets.
const handles = await page.$$('.product-card');
const products = [];
for (const handle of handles) {
try {
products.push(await handle.evaluate(card => ({
name: card.querySelector('.name')?.textContent?.trim() ?? '',
price: card.querySelector('.price')?.textContent?.trim() ?? ''
})));
} finally {
await handle.dispose();
}
}This pattern is useful when each item requires a click, a separate wait, or isolated error handling. It is slower than one bulk callback when all values are already in the DOM.
Use $eval for one required element
const title = await page.$eval('h1', el => el.textContent?.trim() ?? '');Because a missing heading is exceptional in this example, the thrown error is useful. If absence is normal, use page.$() and test the returned handle, or use $$eval and accept an empty result.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteScrape nested fields, links and attributes
Keep DOM reads inside the page callback and return a JSON-compatible structure. Optional chaining prevents a missing badge or image from aborting the entire batch.
const articles = await page.$$eval('article[data-id]', nodes =>
nodes.map(article => ({
id: article.dataset.id ?? null,
title: article.querySelector('h2')?.textContent?.trim() ?? '',
author: article.querySelector('[rel="author"]')?.textContent?.trim() ?? '',
image: article.querySelector('img')?.currentSrc ?? null,
tags: [...article.querySelectorAll('.tag')]
.map(tag => tag.textContent?.trim() ?? '')
.filter(Boolean)
}))
);Do not return DOM nodes, ElementHandles or functions as your final dataset. They are tied to the page and cannot be reliably serialized. Extract the primitive values you need while the callback is running.
Pagination and repeated pages
For a “next” button, extract the current page, then wait for a change that proves navigation completed. A stable item count, URL change, or loading indicator is better than an arbitrary delay.
const all = [];
for (;;) {
await page.waitForSelector('.product-card', { visible: true, timeout: 15_000 });
all.push(...await page.$$eval('.product-card', cards =>
cards.map(card => ({
name: card.querySelector('.name')?.textContent?.trim() ?? '',
href: card.querySelector('a')?.href ?? null
}))
));
const next = await page.$('button.next:not([disabled])');
if (!next) break;
await Promise.all([
page.waitForNavigation({ waitUntil: 'domcontentloaded', timeout: 30_000 }).catch(() => null),
next.click()
]);
}Some applications update the DOM without navigation. In that case, capture the old first-item identifier, click, and wait until it changes with page.waitForFunction or a selector state change.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minutePerformance, reliability and responsible collection
- Extract in one
$$evalcall when possible; repeated round trips between Node.js and the browser add overhead. - Use bounded navigation and selector timeouts, and include the URL and selector in your error log.
- Process very large datasets in pages or batches rather than retaining every handle and every page in memory.
- Reuse a browser for related URLs, but create separate pages when isolation is required.
- Treat an empty
$$result as meaningful. It can indicate a legitimate empty list, a changed selector, an authentication wall or content that has not rendered. - Respect the target site’s terms, robots guidance, authentication rules, rate limits and applicable law. Puppeteer behavior does not grant permission to collect restricted data.
Troubleshooting common failures
“Waiting for selector failed”
The selector may be wrong, content may be inside an iframe, the request may have failed, or the page may require authentication. Confirm the final URL with page.url(), save a screenshot or HTML snapshot for diagnosis, and inspect the selector in browser developer tools. For an iframe, obtain the relevant frame and run its extraction there.
The array is empty
$$eval and $$ return no matches rather than throwing. Check that you waited for the actual data selector, not only a wrapper, and verify that a cookie gate, bot check or “load more” action is not hiding the content.
$eval throws “failed to find element”
The selector has no match at evaluation time. Use waitForSelector when the element should appear, or switch to $() if absence is an expected branch.
Text is blank or incomplete
The visible value may be in a child node, an attribute, a pseudo-element, or a shadow root. Inspect the rendered DOM, choose the correct descendant or attribute, and wait for the component’s populated state. For shadow DOM, evaluate from the host’s shadowRoot when the site exposes one.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Best Value
Navigation hangs or times out
Set an explicit timeout, choose an appropriate waitUntil event, and distinguish a document load from an application becoming ready. Log the response status and final URL where practical; do not solve every timeout with an unlimited wait.
Or skip the browser setup
If your goal is a clean image or PDF of a page rather than DOM-level field extraction, ScreenshotNeo provides a single screenshot API request. It accepts cookie and consent banners as a visitor and removes more than 60 known consent platforms, newsletter popups and chat widgets before capture; each step can be disabled. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed, and response headers identify the page verdict and billing status. Its MCP server exposes take_screenshot, get_page_info and capture_pdf to Claude, Cursor and other MCP clients.
See the complete parameter reference in the ScreenshotNeo documentation.
cURL
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);The free plan includes 1,000 screenshots each month without a card. Paid plans start at $5 for 3,000 shots; every feature is included on every plan. Create a free ScreenshotNeo account.
FAQ
Frequently Asked Questions
Can Puppeteer scrape elements inside an iframe?
Not from the top-level page context. Locate the frame, wait for its selector, and run the same $$eval or $eval operation on that Frame object.
Should I use a delay instead of waitForSelector?
A selector wait is usually more reliable because it ends when the required state exists. A short delay can supplement it for animations, but it should not replace a bounded readiness condition.
Can I return a DOM element from $$eval?
Return serializable text, numbers, booleans, arrays or objects instead. DOM nodes belong to the browser context and are not a durable Node.js result.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
