Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Handle infinite scroll in Node.js differently depending on whether you own the page. For an application you control, place a sentinel after the current results and use IntersectionObserver to request the next page or cursor. For a third-party page, use Playwright to scroll the page’s real scroll container and wait for observable progress—such as a new item, a request, a result count, or an end marker—instead of trusting a fixed delay or a guessed scrollHeight.
Choose the right implementation model
Infinite scrolling is a loading and presentation pattern, not proof that a dataset is endless. The browser may display only the records fetched so far, while the server tracks a page number, opaque cursor, or another pagination state.
| Question | Page you own | Third-party page you automate |
|---|---|---|
| Control | You control the markup, data API and loading state. | You must infer behavior from the rendered page and permitted network activity. |
| Trigger | A sentinel entering the viewport or a scroll container. | Incremental browser scrolling followed by a target-specific wait. |
| Data access | An explicit page number or application-defined cursor. | Rendered DOM, or a documented endpoint if one is available and allowed. |
| Completion | The server says there are no more records. | An explicit end marker, or a bounded sequence of attempts with no progress. |
| Synchronization | Request and state transitions with an in-flight guard. | Observable progress with finite timeouts, not an arbitrary sleep. |
Implement infinite loading when you own the page
Keep pagination state explicit
Define the request contract before writing the observer. A response should identify the items and whether another batch exists, for example with hasMore and an opaque nextCursor. Treat these states separately:
- Request in progress: do not start another request for the same cursor.
- More data available: append the batch and retain or move the sentinel.
- No more data: stop observing and render an end message.
- Request failed: preserve the current list and offer a retry.
Do not derive the cursor from the number of DOM children unless your API explicitly defines that contract. The server should remain the source of truth.
#1 Best Overall
Use a sentinel and IntersectionObserver
Place a small element after the list. The observer asynchronously reports when that element enters the viewport or a specified ancestor. A positive rootMargin starts loading before the user reaches the exact end, giving the request time to finish.
const list = document.querySelector('#results');
const sentinel = document.querySelector('#load-more');
const status = document.querySelector('#status');
let nextCursor = null;
let loading = false;
let hasMore = true;
function appendItems(items) {
for (const item of items) {
const article = document.createElement('article');
article.dataset.id = item.id;
article.textContent = item.title;
list.append(article);
}
}
async function loadNextBatch() {
if (loading || !hasMore) return;
loading = true;
status.textContent = 'Loading…';
try {
const query = nextCursor
? `?cursor=${encodeURIComponent(nextCursor)}`
: '';
const response = await fetch(`/api/results${query}`);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const payload = await response.json();
appendItems(payload.items);
nextCursor = payload.nextCursor ?? null;
hasMore = payload.hasMore === true;
if (!hasMore) {
observer.unobserve(sentinel);
sentinel.hidden = true;
status.textContent = 'End of results';
} else {
status.textContent = '';
}
} catch (error) {
status.textContent = 'Could not load more. Try again.';
console.error(error);
} finally {
loading = false;
}
}
const observer = new IntersectionObserver(
entries => {
if (entries.some(entry => entry.isIntersecting)) loadNextBatch();
},
{ root: null, rootMargin: '400px 0px', threshold: 0 }
);
observer.observe(sentinel);
loadNextBatch();The loading guard matters because an observer callback can run repeatedly while the sentinel remains visible. Without it, concurrent requests can duplicate records or advance the cursor incorrectly. Keep the callback short; the network request and rendering work happen outside the observer’s immediate decision.
Use a nested scroll container when appropriate
If the list scrolls inside a panel rather than the document, pass that element as root:
const panel = document.querySelector('.results-panel');
const observer = new IntersectionObserver(onIntersect, {
root: panel,
rootMargin: '200px 0px',
threshold: 0
});The sentinel must be inside the same scrolling ancestor. A viewport observer will not reliably represent visibility inside an independently scrolling panel.
Render an accessible end and error state
Do not leave users with a permanently spinning loader. Announce loading and errors in a status element, provide a retry control after a failure, and make the terminal state explicit. If the list has a finite result set, offer a conventional pagination or “show more” fallback for keyboard and assistive-technology users.
Automate a third-party infinite-scroll page with Node.js
Install and launch Playwright
A real browser is usually required because the page’s JavaScript creates the requests and appends the records. Install Playwright in your project, install the browser binary it needs, then use a script such as this. The selectors below are deliberately placeholders: inspect the target page and replace them with its actual structure.
Rank #2
import { chromium } from 'playwright';
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage({ viewport: { width: 1280, height: 900 } });
try {
await page.goto('https://example.com/feed', {
waitUntil: 'domcontentloaded',
timeout: 30_000
});
// scrolling and extraction go here
} finally {
await browser.close();
}Identify the actual scroll container
First determine whether the document scrolls or a nested element does. In browser developer tools, scroll the feed and watch which element’s scrollTop changes. A nested container often has overflow-y: auto or scroll and a constrained height.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Do not assume window.scrollTo() reaches the feed. For a document-scrolling page, use window.scrollBy. For a panel, evaluate element.scrollTop = element.scrollHeight on that panel.
Wait for observable progress
The most reliable completion signal is specific to the target. Useful observables include a larger result count, a new item identifier, a response from a known request, or a changed “end of results” marker. This example uses item count and a bounded no-progress counter:
import { chromium } from 'playwright';
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
const itemSelector = '[data-item-id]';
const endSelector = '.end-of-results';
const maxNoProgress = 3;
let noProgress = 0;
try {
await page.goto('https://example.com/feed', {
waitUntil: 'domcontentloaded',
timeout: 30_000
});
for (;;) {
const before = await page.locator(itemSelector).count();
if (await page.locator(endSelector).isVisible().catch(() => false)) break;
await page.evaluate(() => {
window.scrollBy(0, Math.max(window.innerHeight * 0.8, 500));
});
let progressed = false;
try {
await page.waitForFunction(
({ itemSelector, before }) =>
document.querySelectorAll(itemSelector).length > before,
{ itemSelector, before },
{ timeout: 10_000 }
);
progressed = true;
} catch {
// No new item appeared during this attempt.
}
const after = await page.locator(itemSelector).count();
if (progressed || after > before) {
noProgress = 0;
} else {
noProgress += 1;
if (noProgress >= maxNoProgress) break;
}
}
const records = await page.locator(itemSelector).evaluateAll(nodes =>
nodes.map(node => ({
id: node.getAttribute('data-item-id'),
text: node.textContent?.trim() ?? ''
}))
);
console.log(JSON.stringify(records, null, 2));
} finally {
await browser.close();
}For a nested panel, replace the page-level scroll with an evaluation against the inspected container:
await page.locator('.results-panel').evaluate(element => {
element.scrollTop = element.scrollHeight;
});Then wait for a count, identifier, response, or end marker that belongs to that panel. If item nodes are virtualized and old rows are removed, a total DOM count may not increase; track unique IDs or capture each batch as it appears instead.
Wait for requests when the network is the best signal
If inspection reveals a stable request that returns the next batch, wait for that response around the scroll action:
Rank #3
const nextBatch = page.waitForResponse(response =>
response.url().includes('/api/') && response.ok()
);
await page.evaluate(() => window.scrollBy(0, 700));
const response = await nextBatch;
const payload = await response.json();Only use a URL or response shape you actually observed. Do not guess an endpoint, cursor format, authentication header, or selector for an unrelated site. If an official endpoint is documented and your use is permitted, consuming its pages or cursor is often simpler and less fragile than parsing rendered markup.
Know when loading is complete
Prefer an explicit terminal marker
A server-provided hasMore: false, disabled “load more” control, or visible end marker is stronger evidence than geometry. Stop observing or scrolling as soon as that state is authoritative.
Bound no-progress attempts
Some pages do not expose an end marker. After each scroll, compare a target-specific signal. Stop after a finite number of consecutive attempts with no new item, request, or state change. This prevents a stalled page from running forever while still allowing delayed batches.
Do not use stable scrollHeight as the only rule
A temporarily unchanged scrollHeight can mean a request is still pending, content is rendered in a nested container, or lazy resources will change layout later. Conversely, a page can stop changing height while replacing virtualized rows. Treat height as a diagnostic, not completion proof.
Infinite pagination versus lazy loading
Pagination adds records to a feed. Lazy loading defers offscreen resources such as images or frames. One page can use both: a new record may appear immediately while its image loads later. The document’s load event is therefore not proof that every image or feed item is ready. Wait for the particular item, attribute, request, or visual state your task needs.
Performance and reliability practices
For pages you own
- Use
IntersectionObserverinstead of measuring many elements on every scroll event. - Choose a modest positive
rootMarginso slow requests begin before the sentinel is visible. - Abort or ignore stale requests when filters, sorting, or route changes invalidate a cursor.
- Deduplicate records by stable ID before appending.
- Keep the observer callback and scroll handlers cheap; move expensive work to asynchronous code.
For automation
- Use finite navigation, selector, response, and progress timeouts.
- Scroll in increments rather than jumping repeatedly to an assumed absolute bottom.
- Record the signal that caused each batch to be accepted, which makes stalled runs diagnosable.
- Use a bounded attempt count and preserve partial results when the page fails.
- Respect the site’s terms, access controls and rate limits.
If a scroll handler is unavoidable, throttle it with a measured timeout. Scheduling work with requestAnimationFrame alone does not necessarily reduce scroll-handler frequency because animation frames can run at the same rate as scroll events.
Rank #4
Troubleshooting common failures
No new items appear after scrolling
Check whether the feed is inside a nested container, whether a consent dialog or login wall blocks interaction, and whether the page requires a user gesture. Inspect the DOM and network activity before changing the timeout. Then scroll the correct element and wait for the target’s actual signal.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteThe script hangs at the bottom
Add a finite timeout to every wait and a no-progress limit. Look for a disabled control or end marker. If neither exists, stop after a documented number of unchanged attempts and save the records already collected.
Items are duplicated
On an owned page, enforce the in-flight guard and deduplicate by record ID. In automation, the page may re-render existing nodes; collect stable IDs rather than concatenating text from every iteration.
The count never increases
The page may virtualize rows, append inside a different list, or replace a loading placeholder. Track unique identifiers, a known request, or a state attribute instead of total DOM count.
Images are missing
That is often lazy loading rather than pagination. Wait for the image’s required attribute or network response after the record appears; do not infer feed completion from the document’s load event.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Requests fail or return an access challenge
Do not bypass access controls. Verify that your session, permissions and rate are authorized for the site. If a documented API exists, use it under its published rules instead of scraping an interactive page.
Or skip the browser setup
For a one-off page image or a repeatable capture pipeline, ScreenshotNeo provides a GET endpoint and an MCP server. It accepts cookie and consent banners as a visitor, then removes more than 60 known consent platforms, newsletter popups and chat widgets before the shot; each step can be disabled. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed, and response headers identify the page verdict and billing result. Its MCP tools—take_screenshot, get_page_info and capture_pdf—work with Claude, Cursor and other MCP clients.
One-call cURL example (see the ScreenshotNeo documentation):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);ScreenshotNeo includes full-page capture with lazy images loaded, CSS-selector element capture, custom JavaScript and CSS, waits for selectors, delays or network idle, request blocking, cookies and headers, device presets, viewport and retina settings, PDF output, caching, signed links, asynchronous webhooks, bulk capture of up to 100 URLs per call and a usage API. The Free plan includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots. Sign up free.
Recommended Free Tools
Frequently Asked Questions
Should I scrape an infinite-scroll page through its API instead of the DOM?
Use an official, permitted endpoint when one is documented and suitable for your task; it usually exposes pagination more clearly than rendered markup. Otherwise, automate the browser and base completion on the page’s observed state.
Can IntersectionObserver tell me the exact number of visible pixels?
No. It reports threshold and intersection changes, so use it for decisions such as “sentinel entered the root,” not precise overlap measurement.
Is a fixed delay ever useful?
A short delay can help diagnose a page, but it is not a dependable synchronization rule across network conditions. Pair any delay with a specific observable and a timeout.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
