Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →If Puppeteer scrolls without loading more results, targets the wrong area of the page, or waits forever, synchronize the loop with the page’s actual scrolling element and a visible signal of progress. Scroll, then wait for something meaningful to change—such as a larger item count or a loader disappearing—and stop after a defined number of rounds or when the site signals there are no more results.
Why infinite-scroll scripts stall
Infinite scrolling is not one universal browser behavior. A page may load results when the document reaches a threshold, when an inner panel scrolls, or when a loading element enters view. Some interfaces recycle a small set of DOM nodes as you move through a much larger result set. In those cases, the document’s height or the number of rendered elements may not increase even though the list is advancing.
As an Amazon Associate I earn from qualifying purchases.
Puppeteer provides the building blocks, not a universal infinite-scroll loop. page.evaluate runs a function in the page context and awaits a returned Promise; page.waitForFunction waits until a page-context predicate becomes truthy. Use the wait that matches the outcome you need to observe. See the Puppeteer evaluate API and the Puppeteer Page API documentation.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsA reliable debugging sequence
- Find the scrolling element. Determine whether the page document or a nested panel actually scrolls. Inspect the page’s structure and scroll behavior rather than assuming the document owns it.
- Choose a progress signal. Examples include a larger count of rendered items, a new item ID, a changed scroll position, or a known loader appearing or disappearing. For virtualized lists, item count and document height may be misleading; prefer a stable item identity or a site-specific state.
- Scroll the target. Move it by a controlled amount or to its current end, depending on how the site triggers loading.
- Wait for the expected change. Use a predicate or a locator wait for the relevant state rather than assuming the scroll command means loading has finished. Puppeteer locators can wait for an element to be present and in the right state for an action; see Puppeteer page interactions.
- Bound the loop. Set a maximum number of rounds or elapsed time, track consecutive rounds without progress, and stop when the site exposes a real end-of-results condition.
- Investigate no-progress rounds. Recheck the target selector and trigger behavior, then inspect page errors and failed or blocked requests. Increasing a sleep without knowing what should change often only makes a broken loop slower.
Example: scroll and wait for more rendered items
This CommonJS example assumes the result count grows when more content loads and that the target is either the document or a scrollable element selected by CSS. Replace .result-card and, if needed, .results-panel with selectors from the target page. It is an illustrative pattern, not a universal scraper.
#1 Best Overall
async function collectByScrolling(page, {
itemSelector,
scrollTargetSelector = null,
maxRounds = 30,
noProgressLimit = 3,
}) {
let noProgress = 0;
const items = new Set();
for (let round = 0; round < maxRounds && noProgress < noProgressLimit; round++) {
const before = await page.$$eval(itemSelector, els =>
els.map(el => el.textContent?.trim()).filter(Boolean)
);
before.forEach(item => items.add(item));
const previousCount = before.length;
await page.evaluate((selector) => {
const target = selector
? document.querySelector(selector)
: document.scrollingElement;
if (!target) throw new Error('Scroll target not found');
target.scrollTop = target.scrollHeight;
}, scrollTargetSelector);
try {
await page.waitForFunction(
(selector, count) => document.querySelectorAll(selector).length > count,
{ timeout: 5000 },
itemSelector,
previousCount,
);
noProgress = 0;
} catch {
noProgress++;
}
}
return [...items];
}
// Example call:
const results = await collectByScrolling(page, {
itemSelector: '.result-card',
scrollTargetSelector: '.results-panel',
maxRounds: 30,
noProgressLimit: 3,
});The timeout is a per-round bound, not a guarantee that every site will load within five seconds. Adjust it for the page and environment, and keep the overall round or time budget finite. The example adds text to a Set to avoid returning duplicate strings, but production collection should deduplicate with a stable item ID or URL where available. If the page recycles DOM nodes, a count-increase predicate can fail even while the visible results change; wait for a new identity, a changed cursor, or another signal that reflects that interface.
Choose a progress signal that matches the page
When the document scrolls
Use the document’s scrolling element as the target and observe a page-specific result or loading state after scrolling. A changed document height can be useful on a conventional list, but it is not a sufficient end condition by itself.
When a nested panel scrolls
Pass the panel’s selector and scroll that element. Scrolling window or the document while a nested element owns the scroll may do nothing useful. Confirm the panel exists before using it and verify that its scroll position changes.
Free tools Windows power users keep installed
One-click scans. No signup required.
When the list is virtualized
Virtualized interfaces may reuse the same DOM rows, so rendered-item count can stay constant. Track stable identifiers, item text plus a distinguishing field, a page-specific cursor, or another observable transition. Do not assume a fixed number of nodes means the results have stopped.
When loading is triggered by a threshold or intersection
Setting the target directly to its current end works only if the page responds to that action. Some pages require incremental movement or a particular sentinel entering view. If the target scrolls but nothing changes, test how the page triggers loading and adapt the movement to that behavior.
Wait for a transition, not mere presence
A selector wait can return immediately if the selector already exists. If a loader or result container is present from earlier content, waiting only for its presence does not prove that the next batch arrived. Wait for a changed count, a new item identity, a changed state, or the loader to disappear after it appears. page.waitForFunction is useful when the expected condition is a JavaScript predicate; locator waits are useful for element presence and actionability.
Fixed delays are easy to add but can finish before a slow response or waste time when a fast response has already completed. Prefer a condition that reflects the page’s actual progress. If the page has no observable signal, use a bounded delay only as a fallback, with a finite retry budget and diagnostics for no-progress rounds.
Common failures and fixes
- Scrolling happens, but no content loads: Confirm the real scroll owner, selector, and loading trigger. Check whether the page requires reaching a threshold or bringing a sentinel into view.
- The script collects old content: Wait for a meaningful change rather than for scrolling to finish. A predicate can compare item counts, IDs, or state before and after the action.
- The wait succeeds immediately: The selector may already exist. Wait for a transition—such as count growth or a new item identity—instead of simple presence.
- The loop runs indefinitely: Add a site-specific end condition, a no-progress limit, and a maximum iteration or elapsed-time budget. Infinite scrolling may have no finite bottom.
- The page height stops changing: Check for nested scrolling, virtualized rows, or threshold-based loading. Height is only a useful signal when the page’s behavior makes it one.
- The scroll target is missing: Check the selector and page state before scrolling. The page may not have reached the expected state, or a login or consent screen may be showing instead of the results.
Or skip the browser setup
If you need a clean image or PDF of a page rather than a Puppeteer scraping loop, ScreenshotNeo can capture it with one GET request. Its API accepts a URL and returns a PNG, JPEG, WebP, or PDF. See the ScreenshotNeo API documentation.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpScreenshotNeo accepts cookie or consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, with verdict and billing information in response headers. Its MCP server provides take_screenshot, get_page_info, and capture_pdf tools for AI agents. The Free plan includes 1,000 shots per month without a card; paid plans start at $5 for 3,000 shots.
Sign up free for 1,000 screenshots a month—no card required.
Frequently Asked Questions
Does Puppeteer provide a built-in infinite-scroll loop?
No. Puppeteer provides page evaluation, wait, and interaction APIs; the loop and its stopping condition must match the target site.
What if the page does not have a finite bottom?
Stop on a site-specific end-of-results signal, or enforce finite iteration and elapsed-time budgets with a no-progress limit.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
