Use a JavaScript-capable browser, scroll the element that actually contains the product list, and wait for new cards to appear after each scroll. Collect a stable product ID or URL, deduplicate results, and stop only when the site signals the end or several properly waited scroll cycles add nothing new. This is a practical stopping rule—not proof that every store has exposed its entire catalog.
First identify how the store reveals products
Infinite scroll is only one way a store can reveal a larger result set. Before automating, determine whether the page scrolls normally, the product panel scrolls independently, a “Load more” button appends products, or the catalog is split across pages. These patterns need different actions. Google describes pagination, load more, and infinite scroll as distinct approaches to showing subsets of ecommerce results (Google Search Central guidance).
As an Amazon Associate I earn from qualifying purchases.
- Infinite scroll: scroll the page or listing container and wait for more cards.
- Load more: click the button and wait for appended cards.
- Pagination: follow the next-page link and collect each page.
A static HTML request may contain only the first batch when later cards depend on JavaScript or user interaction. Google notes that its crawler generally does not click buttons or trigger user-action JavaScript; crawlable sequential links can expose paginated pages differently.
Recommended Free Tools
Capture an infinite list with Playwright
The example below is a starting pattern, not a store-specific tested scraper. Install Playwright for your chosen language, open the target page, and replace the card and bottom-sentinel selectors with ones verified in that page’s markup. The code records product links as identifiers; if a store exposes a more stable product ID, use it instead.
#1 Best Overall
const cards = page.locator("YOUR_PRODUCT_CARD_SELECTOR");
const seen = new Set();
let unchangedRounds = 0;
while (unchangedRounds < 3) {
// Read identifiers from the cards currently rendered.
const ids = await cards.locator("a[href]").evaluateAll(links =>
links.map(a => a.href)
);
const before = seen.size;
for (const id of ids) seen.add(id);
// Use a verified bottom sentinel or listing element.
await page.getByText("YOUR_BOTTOM_SENTINEL").scrollIntoViewIfNeeded();
await page.waitForTimeout(750);
const nextIds = await cards.locator("a[href]").evaluateAll(links =>
links.map(a => a.href)
);
for (const id of nextIds) seen.add(id);
unchangedRounds = seen.size === before ? unchangedRounds + 1 : 0;
}
console.log([...seen]);Replace the fixed 750 ms pause with a site-specific wait when possible—for example, wait for a loading indicator to disappear or for the card count or identifiers to change. A fixed delay can be too short on a slow response and unnecessarily long on a fast one. Playwright’s input documentation describes scrolling a bottom element into view to trigger an infinite list; for a nested panel, hover it and use the mouse wheel or adjust that container’s scrollTop.
Make the selectors and identifier meaningful
- Inspect a product card and choose a selector that matches cards, not every link elsewhere on the page.
- Choose one canonical product URL or stable product ID per card. Normalize URLs only if you understand which query parameters distinguish variants.
- Use a sentinel near the end of the listing if available. Scrolling a footer or unrelated element may not trigger the product loader.
Do not enumerate a moving list too early
Wait for the relevant load to finish before reading the list. Playwright’s locator documentation notes that locator.all() returns the elements present immediately rather than waiting for a dynamic list to settle, so its results may be unpredictable while the list changes (Playwright locator documentation). The sketch uses evaluateAll to read current links, but it still needs a meaningful wait around each scroll.
Use a completion check, not just a scroll-to-bottom check
A page can extend after each batch arrives, so reaching the current bottom once does not establish that the list is finished. Prefer an explicit end-of-results message or a stable displayed total. Otherwise, collect unique identifiers and stop after several successive scroll-and-wait cycles add none, while checking that the request is not merely delayed or failing. Three unchanged rounds in the example are an operational choice, not an official Playwright or Google guarantee.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minute- Read the currently rendered product identifiers and add unseen ones to a set.
- Scroll the verified list target or click the relevant load control.
- Wait for the site’s loading state or a meaningful card/list change.
- Read identifiers again and reset the no-growth counter if new unique products appeared.
- Stop at a clear site end state, or after repeated no-growth rounds once slow loading and wrong-target issues have been ruled out.
- If the page displays a total, compare it with the unique captured count and investigate any mismatch.
A displayed total is a useful cross-check, not an absolute guarantee: filters, unavailable items, hidden cards, and duplicates can make the visible count and collected count differ.
When scrolling is the wrong method
Load-more controls
Click the control and wait for additional cards or a loading state to finish. Scrolling alone may never request another batch. Confirm that the click appended new products before repeating it.
Paginated catalogs
Follow each sequential next-page link and collect each page’s products, deduplicating across pages. Google recommends crawlable sequential links for paginated content and generally follows URLs in anchor href attributes rather than clicking buttons. That guidance concerns crawling and indexing; it does not guarantee a particular automation workflow can retrieve a store’s full catalog.
Nested scrolling panels
If the document stays still while the product panel moves, target the panel. Playwright’s input guidance covers controlling the wheel or a particular scroll container; applying scrollTop to the container is another direct option when the correct element is known.
Troubleshooting missing or duplicated products
| Symptom | Likely cause | What to check or change |
|---|---|---|
| No new cards after scrolling | The wrong element scrolled, the load is still in flight, or the page uses a button or pagination. | Check which element’s scroll position changes, inspect for a loading state, and identify the page’s actual loading pattern. |
| The script stops after the first batch | The bottom condition may refer to the current document end, not a listing sentinel, or the wait is too short. | Scroll a verified element near the list bottom and wait for a card/count change or loading completion. |
| Product count changes between reads | The list is still being rendered or re-rendered. | Wait for a stable loading signal before enumeration; avoid relying on immediate locator snapshots for a changing list. |
| Collected count exceeds the site’s total | Repeated cards, variants, promotional links, or unstable URL parameters may be counted separately. | Deduplicate on a stable product URL or site ID and inspect the card selector and URL normalization. |
| Collected count is below the displayed total | Some cards may be hidden, unavailable, filtered, or not yet loaded; the selector may omit card types. | Check filters and card markup, wait for loads, and verify whether the displayed total describes the same result set. |
Or skip the browser setup
ScreenshotNeo is a website screenshot API and MCP server. A screenshot captures what a page renders; it does not extract a complete structured product catalog. For inspecting a particular rendered state, one GET request can return an image or PDF, with options such as full-page capture and lazy-image loading. See the ScreenshotNeo API documentation.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpScreenshotNeo accepts cookie or consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing status. Its MCP server offers take_screenshot, get_page_info, and capture_pdf tools for AI agents and MCP clients. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots.
Sign up for 1,000 free screenshots a month, with no card required.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
