Handle Selenium pagination as a sequence of verified page changes: collect the current results, check whether Next is available, click it, wait for an observable signal that the new page has loaded, then locate and collect the new results. The right signal and the stopping condition depend on the site’s markup and behavior; Selenium has no universal pagination command.
Why pagination needs more than a click
A successful click does not prove that the next page’s results are ready. On JavaScript-driven sites, the document can report a completed load while scripts are still updating the interface. Selenium’s waits documentation explains that readyState covers assets defined in the HTML, not necessarily later JavaScript changes. Use a condition-based wait tied to the page transition instead. Selenium: Waiting Strategies
As an Amazon Associate I earn from qualifying purchases.
Pagination may navigate to a new URL, update a page number, replace a results region, or fetch content in place. First inspect the application’s DOM and choose a stable locator for the results, a Next control, and—if available—a page marker or other signal that distinguishes the new state from the old one.
Free tools Windows power users keep installed
One-click scans. No signup required.
A Python pattern for next-page pagination
This illustrative loop uses Selenium’s Python binding and an explicit wait. Replace the CSS selectors and process function with ones that match the target site. It assumes the visible current-page text changes after clicking Next and that the control is a button whose availability is reflected by is_enabled().
#1 Best Overall
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
wait = WebDriverWait(driver, 10)
while True:
# Find the current results afresh and process them before advancing.
rows = driver.find_elements(By.CSS_SELECTOR, "table tbody tr")
for row in rows:
process(row.text)
# Adapt the selector and availability check to the site's actual markup.
next_buttons = driver.find_elements(By.CSS_SELECTOR, "button.next")
if not next_buttons or not next_buttons[0].is_enabled():
break
old_page_text = driver.find_element(
By.CSS_SELECTOR, ".current-page"
).text
next_buttons[0].click()
try:
wait.until(
lambda d: d.find_element(
By.CSS_SELECTOR, ".current-page"
).text != old_page_text
)
except TimeoutException as exc:
raise TimeoutException(
f"Pagination did not leave page {old_page_text!r} "
"within 10 seconds"
) from excThe timeout is deliberate: if the expected transition does not occur, the script reports it rather than silently treating the same page as the next one. Choose a timeout appropriate to the application, and make the failure message useful to whoever runs the scraper or test.
Adapt the end condition
The sample stops when Next is missing or disabled. That is only correct if the site represents its final page that way. Other applications expose a final-page marker, a known page count, an unavailable state in an attribute, or numbered navigation. Inspect the actual control and use its real end signal; a disabled-looking button may not report disabled through Selenium’s is_enabled().
Rank #2
Choose a transition signal that proves progress
Use the strongest observable change the page provides:
Recommended Free Tools
- Page marker: wait for the current-page label to differ from its previous value. This is specific when the label updates reliably.
- URL: for navigation that changes the address, wait for the expected URL or URL change.
- Results: wait for a known result value or the result text to change. Merely waiting for any row to exist is weak if rows from the previous page remain visible.
- Replaced results region: save the old region and wait for it to become stale, then locate the new region and its rows. Selenium documents staleness as an Expected Condition; the Python API reference includes
staleness_of. Selenium: Waiting with Expected Conditions · Python Expected Conditions API, Selenium 4.50.0 - Loading indicator: wait for the site’s specific loading state to finish, ideally alongside a signal that the intended page changed.
No one signal fits every application. A page marker or expected URL is meaningful only if that site updates it consistently; after a rerender, reacquire elements by locator instead of assuming their old references remain valid.
Build a reliable pagination loop
- Inspect the page. Identify stable locators for the results, Next control, and a transition signal. Check how the site marks the final page and whether it updates the URL or replaces the results container.
- Process the current page first. Locate the current results and extract the information you need before clicking onward.
- Check the end condition. Stop if the site’s actual signal says there is no next page. Do not click a missing, disabled, or unavailable control.
- Click Next once. Avoid repeated clicks while a transition is in progress.
- Wait for evidence of the transition. Use an explicit wait for the page-specific signal. Selenium defines explicit waits as polling for a specified condition until it becomes true or the timeout expires. Selenium: Waiting Strategies
- Re-find and process. Locate the new results and pagination controls after the change; a rerender can invalidate existing element references.
- Bound the run. Use a maximum page count or track visited page markers or URLs when an unexpected transition could send the loop around again.
Waits: what to use and what to avoid
Prefer explicit, condition-based waits
An explicit wait polls for a particular condition, such as a changed page number, a visible element, or a stale old results region. Selenium’s documentation describes this as waiting until the specified condition evaluates true or the timeout expires. The condition should demonstrate the transition you need, not just that some element exists somewhere on the page. Waiting Strategies
Do not use fixed sleeps as the main synchronization method
A hard-coded sleep can be too short when the site is slow and unnecessarily long when the page responds quickly. It also does not tell you whether the expected page state was reached. Use a sleep only when a specific application behavior genuinely requires a fixed delay, not as a substitute for checking the result.
Avoid mixing implicit and explicit waits casually
An implicit wait affects element-location calls across the session. Selenium warns that combining implicit and explicit waits can produce unpredictable timing, with total waits exceeding the apparent timeout. Prefer a consistent wait strategy and follow the guidance for the Selenium binding and version used by your project. Waiting Strategies
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Check binding and version differences
The code above is Python-specific. Selenium’s Expected Conditions guide notes that .NET stopped supporting its Expected Conditions classes in Selenium 4, so Python or Java examples should not be copied into .NET without checking that binding’s API. The linked Python API reference is for Selenium 4.50.0; verify condition names and behavior against the version installed in your project. Expected Conditions · Python API reference
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Common pagination failures and fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Selenium reads the same results after clicking Next. | The click happened, but the script read before the asynchronous update completed, or its wait condition was too weak. | Wait for a changed page marker, expected URL, changed result value, or stale old container before reading again. |
StaleElementReferenceException after navigation. |
The application replaced the element or its parent during rerender. | Wait for the old region to become stale if appropriate, then locate the new region and controls using their locators. |
| The loop stops before the last page, or clicks a disabled Next control. | The site’s disabled or end state is represented differently from the sample’s assumptions. | Inspect the control’s DOM and state attributes; implement the site’s actual final-page condition rather than relying only on is_enabled(). |
| The wait times out even though the page changed. | The selected signal is not updated by that transition, or the locator no longer matches after rerender. | Inspect the resulting DOM and choose a reliable signal for that application, such as the URL or replacement of the results region. |
| The loop repeats a page or runs indefinitely. | The control did not advance, the transition signal was not unique, or unexpected navigation returned to a visited state. | Track page markers or URLs and set a maximum page count; raise an error when progress stops unexpectedly. |
| A wait seems to take longer than its configured timeout. | Implicit and explicit waits may be interacting. | Review the session’s implicit-wait setting and avoid combining wait styles without understanding the binding’s timing behavior. |
Or skip the browser setup
If the task is to capture page screenshots rather than automate extraction from each result, ScreenshotNeo provides a screenshot API and MCP server. A single GET request can return a screenshot or PDF; this does not replace Selenium when you need to inspect or process paginated records.
For example, using cURL to capture a page:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo documentation for request options. Cookie banners, popups, and chat widgets are removed before capture; bot checks, blank pages, and failed loads are not billed. Its MCP server lets AI agents take screenshots. The free plan includes 1,000 screenshots a month with no card, and paid plans start at $5 for 3,000. Sign up for free.
Frequently Asked Questions
Does Selenium have a built-in pagination method?
No. Pagination controls and their update behavior are part of each application, so the script must locate the control, wait for that site’s transition signal, and implement its end condition.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Why is waiting for rows to exist not enough?
Rows may still be present from the previous page. A changed page marker, expected URL, changed result value, or replaced results container provides stronger evidence that the intended transition occurred.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
