To handle infinite scroll in Python, scroll the element that actually drives loading, wait for a meaningful page change, collect the new items, and stop only when the page provides a completion signal or repeated attempts make no progress. Scrolling to the document bottom once is not a reliable way to load every result: the page may use a nested scroll container, a sentinel near the end of the list, or another site-specific trigger.
This guide uses Playwright’s Python API for the main example and explains how to adapt the same approach to Selenium. The selectors and completion signals must be chosen for the page you are automating.
How infinite scroll works in browser automation
Infinite scroll is a browser interaction pattern, not a special Python feature. A page loads an initial batch of content, then adds more when a trigger—often a scrolling threshold or an element entering view—is reached. The automation must reproduce the trigger and observe whether the page changed.
First identify what scrolls. It might be the document window, a nested panel such as a results list, or a target element near the current end of the content. Playwright documents scrolling an element into view to trigger an infinite list, mouse-wheel scrolling, and changing a chosen container’s scrollTop: Playwright input actions.
#1 Best Overall
Next identify observable evidence of loading: a new item appears, the item count rises, a loading indicator disappears, or an end-of-results marker becomes visible. Navigation finishing does not mean that future batches have loaded. Selenium’s wait guidance notes that elements can load at different times, and Playwright provides locator-based state waits.
Choose a scroll target and a stop condition
Document scrolling
Use document scrolling when the page itself moves as you scroll. A reliable way to trigger many sentinel-based lists is to bring the last currently visible item or a footer-like target into view, rather than jumping to the absolute bottom once. The page may need several such interactions before it adds another batch.
Nested scroll container
If a list or panel has its own scrollbar, scrolling the document may do nothing. Inspect the page in a browser’s developer tools and identify the element whose scrollTop changes as the list moves. Scroll that element, or use the mouse wheel while the pointer is over it.
Site-specific completion signals
Prefer an explicit end marker, a disabled or absent “Load more” control, or another documented page state. If no explicit marker exists, compare the number or stable identifiers of collected items across attempts and stop after a bounded number of rounds with no new results. No single selector or stop condition works for every site.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Playwright Python: a bounded infinite-scroll loop
Install Playwright and its browser binaries for your environment, then adapt the selectors and URL below to the page you are permitted to access. The example scrolls the last matching item into view, waits for the item count to increase, and tolerates a limited number of no-progress rounds. It is a pattern to adapt, not a claim that these selectors fit every site.
from playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeoutError
URL = "https://example.com/results"
ITEMS = "article.result" # Replace with the site's item selector
END_MARKER = "text=No more results" # Replace, or set to None if unavailable
MAX_STALLED_ROUNDS = 3
LOAD_TIMEOUT_MS = 5000
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(URL, wait_until="domcontentloaded")
stalled_rounds = 0
seen = set()
while stalled_rounds < MAX_STALLED_ROUNDS:
items = page.locator(ITEMS)
before = items.count()
if before:
# Bring the current tail into view to trigger pages that use a sentinel.
items.nth(before - 1).scroll_into_view_if_needed()
else:
# Initial state: move the document down to prompt the first batch.
page.mouse.wheel(0, 700)
try:
page.locator(ITEMS).nth(before).wait_for(
state="visible", timeout=LOAD_TIMEOUT_MS
)
stalled_rounds = 0
except PlaywrightTimeoutError:
stalled_rounds += 1
# Re-count after the wait; dynamic lists can change while loading.
items = page.locator(ITEMS)
after = items.count()
for i in range(after):
item = items.nth(i)
# Choose a stable key for the site, such as a record ID or link.
key = item.get_attribute("data-id") or item.inner_text()
if key not in seen:
seen.add(key)
print(key)
if END_MARKER and page.locator(END_MARKER).count() > 0:
break
if after <= before:
stalled_rounds += 1
else:
stalled_rounds = 0
browser.close()For real collection, replace the print with writing records to your database or an output file. Choose a stable identifier: text alone can collide or change, while a page-specific record ID or canonical link is often more suitable. If the list re-renders its elements, re-query the locator after each interaction instead of retaining stale element handles.
This illustrative loop may count a stalled round both when the wait times out and when the count does not increase. If you want exactly one increment per attempt, centralize the progress check in one place. The important properties are a bounded loop, an observable progress measure, and a site-specific end signal.
Use Playwright locators and waits carefully
Playwright locators re-resolve elements when operations run and provide auto-waiting for relevant actions. Its documentation describes locators as central to auto-waiting and retry behavior: Locator | Playwright Python. Prefer locator operations and explicit conditions over arbitrary pauses where possible.
Free tools Windows power users keep installed
One-click scans. No signup required.
For example, if a known “Load more” button appears, click it and wait for an item count increase:
button = page.get_by_role("button", name="Load more")
old_count = page.locator(ITEMS).count()
button.click()
page.locator(ITEMS).nth(old_count).wait_for(state="visible", timeout=8000)When a dynamic list is changing, do not assume locator.all() waits for the list to stabilize. Playwright explicitly cautions that all() returns immediately and can be unpredictable when matches change. Wait for a meaningful state or count first, then collect: Page | Playwright Python.
Nested container scrolling
Once you have identified the correct container selector, set its scroll position directly when that is more reliable than wheel movement:
container = page.locator("div.results-scroll-area")
container.evaluate("el => el.scrollTop = el.scrollHeight")If the page only loads when the pointer is over the panel, use a mouse-wheel action at a point inside it instead. Playwright documents both wheel input and scrolling a selected container in its input guide.
Recommended Free Tools
Avoid relying on fixed sleeps
A fixed delay can serve as a fallback when a known site has no observable load signal, but elapsed time does not prove that content arrived. If a delay is necessary, follow it with a count, selector, or end-marker check. Playwright’s page reference discourages page.wait_for_selector in favor of locator-based methods; use locator waits for the state you need.
Selenium Python alternative
If your project already uses Selenium, keep its browser setup and use an explicit wait for the page condition rather than assuming a fixed load time. The Selenium Python waits guide explains that elements may load at different times and describes explicit waits: Selenium Python Bindings: Waits. That documentation page is older than the cited Playwright pages; check the API documentation for the Selenium version installed in your project before relying on version-specific details.
A typical Selenium structure is to locate the current tail, scroll it into view, wait for the item count to increase, and repeat with a cap. The exact imports, driver setup, locator syntax, and expected condition depend on the installed Selenium version and page:
# Structural example: adapt selectors and driver setup to your project.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
items_selector = "article.result"
old_count = len(driver.find_elements(By.CSS_SELECTOR, items_selector))
last_item = driver.find_elements(By.CSS_SELECTOR, items_selector)[-1]
driver.execute_script("arguments[0].scrollIntoView({block: 'end'});", last_item)
WebDriverWait(driver, 8).until(
lambda d: len(d.find_elements(By.CSS_SELECTOR, items_selector)) > old_count
)Handle a timeout as a possible no-progress round, not proof that the page is complete. Check for a site-specific end marker and enforce a maximum number of retries so a stalled page cannot keep the script running forever.
Collect without missing or duplicating items
- Re-query after scrolling. A framework may replace list nodes during rendering; old element references can become stale.
- Wait before bulk collection. Confirm the new item or expected state before reading the current list. Playwright warns that immediate bulk listing can be unreliable for changing results.
- Deduplicate by a stable key. Use a record ID, canonical URL, or another unique field rather than assuming each scroll adds only new records.
- Record progress. Log the round, item count, and last stable key. This helps distinguish a genuinely finished list from a broken trigger or selector.
- Persist incrementally. Save each newly observed batch so a later timeout or browser failure does not discard earlier work.
If the page changes existing records as well as appending new ones, count growth alone is not enough. Compare identifiers or relevant fields and update records when their content changes.
Troubleshooting common failures
Scrolling the page does not load anything
Likely cause: the page uses a nested scroll area, or the trigger is a sentinel that has not entered view. Fix: inspect which element’s scroll position changes, then scroll that container or bring the current tail into view.
The script keeps returning only the first batch
Likely cause: it collects immediately after scrolling, or waits for navigation rather than the asynchronous list update. Fix: wait for a specific new item, increased count, or loading-state transition before collecting.
The script loops forever
Likely cause: it has no bounded retry count or ignores the page’s completion state. Fix: cap stalled rounds, check an end marker or disabled load control, and report the final count and stop reason.
Items are duplicated or skipped
Likely cause: batches overlap, list nodes are replaced, or the script treats position as identity. Fix: re-query each round and deduplicate by a stable item identifier. Persist new records incrementally.
A wait times out despite visible content
Likely cause: the chosen selector, expected state, or requested next index does not match the page’s behavior. Fix: verify the selector against current markup and wait for the actual signal—for example, an item count increase, an end marker, or disappearance of a loading indicator.
The list works manually but not in automation
Likely cause: the browser interaction differs from the page’s expected trigger, or the target page presents a different state to the automated session. Fix: use the same scroll region and interaction that works in the browser, inspect what changes after each action, and respect the site’s terms and access controls. Do not attempt to bypass access restrictions.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Reliability, performance, and responsible access
Keep each run bounded and observable: set a maximum number of stalled attempts, record the reason for stopping, and preserve already collected data. Wait only for the state required for the next action; long generic delays slow automation without proving readiness. The appropriate timeout depends on the page and environment, so tune it from observed behavior rather than treating one duration as universal.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Infinite scrolling can make a page load more than intended. Stop once the target end condition is met, and avoid unnecessary repeated requests. Follow the target site’s terms, access controls, and applicable rules for collecting its content.
Or skip the browser setup
If your task is to capture a page as an image or PDF rather than extract every record, ScreenshotNeo is a website screenshot API and MCP server for developers. It is not a replacement for scrolling through and parsing an infinite list when you need every item. Its API can capture a screenshot or PDF with a GET request; see the ScreenshotNeo documentation.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpScreenshotNeo removes known cookie and consent banners, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, with response headers identifying the page verdict and billing status. Its MCP server provides take_screenshot, get_page_info, and capture_pdf tools for AI agents. The Free plan includes 1,000 screenshots per month without a card; paid plans start at $5 for 3,000 shots, and every feature is available on every plan. Sign up for ScreenshotNeo’s free plan.
Frequently Asked Questions
Can I use requests or Beautiful Soup instead of a browser?
They do not perform browser scrolling or execute the page’s client-side interaction. Use them only if the site exposes the needed data through an accessible endpoint or static HTML and you are authorized to access it.
Does scrolling to the bottom guarantee every result has loaded?
No. Pages can use sentinels, nested scroll regions, or other site-specific triggers; verify a page-specific completion signal.
Can ScreenshotNeo return every item from an infinite list?
It is a screenshot and PDF capture service, not a tool for extracting all records from a dynamic list. Use browser automation when you need the underlying items.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
