Recommended Free Tools
Handle an infinite-scroll page in Ruby by repeatedly scrolling the element that owns the feed, waiting for a measurable change, and stopping on a clear end condition. Do not treat document.readyState or a completed navigation as proof that JavaScript-rendered results are finished. A reliable loop has four parts: identify the scroll region, trigger loading, wait for new state, and enforce bounds for time and iterations.
What infinite scroll changes in a Ruby script
A conventional page usually returns a document whose content is present after navigation. An infinite-scroll page appends items later, often after the viewport reaches a threshold or a sentinel element. Network requests, rendering, lazy images and bot checks can all finish at different times. Selenium’s navigation readiness therefore cannot tell you that the feed is complete.
Watir is the Ruby-focused browser automation option in the reviewed material. Its project includes scrolling support; the Watir 6.16 announcement (December 16, 2018) described scrolling as useful for “static css styles, ‘infinite scroll’ pages, and elements inside of scroll bars,” and the Watir 7.2 announcement (December 24, 2022) documented advanced origin-based scrolling. Those are historical release notes, not a promise about the newest installed version. Watir 7.2 listed Selenium 4.2 and Ruby 2.7 as minimum requirements; verify compatibility with your own versions.
Choose the correct scrolling target
Window-owned feeds
Some sites attach the feed to the browser window. In that case, scrolling the page or moving a bottom sentinel into view triggers the next request.
#1 Best Overall
Nested scroll containers
Dashboards, chat-like feeds and result panels often use an element with overflow: auto or overflow: scroll. The window may remain at the top while the panel moves. Inspect the DOM and CSS, then select that element and observe its item count or loading state. Scrolling the wrong target produces a perfectly healthy-looking script that never loads another page.
Find a stable marker
- An item collection such as
.result-cardor[data-testid='result']. - A loading indicator that appears and disappears.
- A “no more results” message.
- A footer or sentinel at the feed boundary.
- A stable identifier on each item for deduplication.
Prefer attributes intended for automation over presentation classes. If the site offers an accessible role or data attribute, use it.
Watir: a bounded infinite-scroll loop
The following example uses Watir and Chrome. It waits for the result count to increase instead of relying on a fixed sleep. Adapt selectors, the end condition and the limits to the target site; there is no universal delay or iteration count.
require "watir"
browser = Watir::Browser.new(:chrome, headless: true)
browser.goto("https://example.test/results")
items = browser.elements(css: ".result-card")
seen = {}
max_rounds = 100
stalled_rounds = 0
max_rounds.times do |round|
# Capture currently visible records before triggering another load.
items.each do |item|
id = item.attribute_value("data-id") || item.text
seen[id] = item.text unless id.nil? || id.empty?
end
break if browser.text.include?("No more results")
before = browser.elements(css: ".result-card").size
sentinel = browser.element(css: ".feed-sentinel")
if sentinel.exists?
sentinel.scroll_into_view
else
browser.execute_script("window.scrollTo(0, document.body.scrollHeight)")
end
changed = Watir::Wait.until(timeout: 20) do
after = browser.elements(css: ".result-card").size
loading_gone = !browser.element(css: ".loading").present?
after > before || (loading_gone && browser.text.include?("No more results"))
end
if changed
stalled_rounds = 0
else
stalled_rounds += 1
break if stalled_rounds >= 3
end
warn "round #{round + 1}: #{browser.elements(css: '.result-card').size} items"
end
puts "unique items: #{seen.length}"
browser.closescroll_into_view is useful when a sentinel is what activates loading. When the page itself owns scrolling, the JavaScript fallback moves the window to the bottom. If the feed is nested, replace the fallback with a scroll on that element and wait on its children:
Free tools Windows power users keep installed
One-click scans. No signup required.
panel = browser.element(css: ".results-panel")
before = panel.elements(css: ".result-card").size
browser.execute_script(<<~JS, panel.element)
const el = arguments[0];
el.scrollTop = el.scrollHeight;
JS
Watir::Wait.until(timeout: 20) do
after = panel.elements(css: ".result-card").size
after > before || panel.text.include?("End of results")
endThe exact JavaScript argument handling can vary with the driver and Watir version. If direct element scrolling is available in your installed Watir release, use that API; otherwise execute a small script against the selected element.
Rank #2
Selenium WebDriver with Ruby
Use Selenium when an existing test suite already depends on it or you need direct WebDriver control. The same design applies: scroll, wait for an observable change, and stop on bounds or a site-specific signal.
require "selenium-webdriver"
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument("--headless=new")
driver = Selenium::WebDriver.for(:chrome, options: options)
wait = Selenium::WebDriver::Wait.new(timeout: 20)
driver.navigate.to("https://example.test/results")
max_rounds = 100
stalled = 0
max_rounds.times do
cards = driver.find_elements(css: ".result-card")
before = cards.length
break if driver.find_element(css: ".no-more").displayed? rescue false
sentinel = driver.find_elements(css: ".feed-sentinel").first
if sentinel
driver.execute_script("arguments[0].scrollIntoView({block: 'end'});", sentinel)
else
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
end
begin
wait.until do
after = driver.find_elements(css: ".result-card").length
after > before || driver.find_elements(css: ".no-more").any?(&:displayed?)
end
stalled = 0
rescue Selenium::WebDriver::Error::TimeoutError
stalled += 1
break if stalled >= 3
end
end
puts driver.find_elements(css: ".result-card").length
driver.quitIn production code, avoid the compact rescue expression used only for illustration and handle NoSuchElementError explicitly. Recreate a fresh wait deadline for each round if your driver’s wait object is not reusable in the way your version expects.
Waiting correctly after each scroll
Wait for a count increase
Counting cards is simple and usually robust. Record the count immediately before scrolling and wait until it increases. It also lets you detect a stalled feed.
Wait for a new identity
Some interfaces recycle DOM nodes, so the count may remain constant while content changes. Capture the last item’s stable ID or text and wait for a different value. Do not use position alone if virtualization is present.
Wait for loading state transitions
A spinner disappearing is useful only when combined with another condition. A request can fail and hide the spinner without adding items. Pair it with a count increase, a new ID, or an explicit end message.
Rank #3
Wait for a network-idle concept carefully
Pages may maintain analytics or polling requests forever, so “network idle” can be unsuitable. A page-specific DOM condition is generally more deterministic for an infinite list.
Stopping without missing results or hanging forever
- Target found: stop as soon as the required record appears.
- End marker: stop on “no more,” a disabled next control, or an API-provided terminal state.
- Repeated stalls: stop after a small number of consecutive waits with no new state, then record a diagnostic.
- Hard bounds: set maximum rounds and an overall deadline. Choose values from the target site, not from a universal rule.
- Duplicate protection: store stable IDs and deduplicate when collecting all records. A page can append duplicates after retries.
When completeness is business-critical, save the final URL, item count, last observed ID, console errors and a screenshot on failure. A bounded partial result with diagnostics is safer than an unbounded process that consumes a worker indefinitely.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallHandling failures and unusual page behavior
The count never changes
Check that the selector matches rendered items, that you are scrolling the owning container, and that the site has not returned a login, consent or bot-check page. Inspect the element’s scrollHeight and clientHeight to confirm it can scroll.
Content appears only after a human-like gesture
Some interfaces require a sentinel to enter the viewport, a click on “Load more,” or a focus event. Implement the documented interaction rather than adding increasingly long sleeps.
Virtualized lists lose earlier nodes
A virtualized UI removes off-screen elements. Persist each record as soon as it appears and deduplicate by a stable key; do not expect the DOM to contain every result at the end.
Rank #4
Lazy images remain blank
Wait for the image’s complete state or a non-empty natural width when images matter. Scrolling can trigger image loading separately from item insertion.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Timeouts and transient failures
Use a retry policy with a cap, capture diagnostics, and distinguish “no new results” from a browser or network exception. Repeating forever can duplicate records and hide a broken session.
Consent banners, authentication and bot checks
Handle consent before measuring the feed, authenticate through the supported test flow, and respect the site’s terms. A CAPTCHA is not an infinite-scroll condition; stop and report it instead of attempting to defeat it.
Performance and reliability practices
- Collect only required fields and persist incrementally for very large feeds.
- Use headless mode in CI, but reproduce failures once with a visible browser.
- Keep selectors narrow; querying the whole document every few milliseconds adds overhead.
- Prefer event-driven waits with a sensible timeout over fixed sleeps. A short sleep can race slow networks; a long sleep wastes every fast iteration.
- Throttle scrolling and requests when the target site is sensitive to rapid automation.
- Log round number, previous and new counts, elapsed time, URL, and the reason for stopping.
Infinite scroll and search indexing are different problems
If you are building the site rather than testing it, browser automation is not the crawlability solution. Google Search Central guidance says infinite-scroll content should support paginated loading: each chunk needs a persistent, unique URL and stable content for that URL. Its lazy-loading guidance says relevant content should load when it becomes visible without depending on a user scrolling or clicking, because Google Search does not interact with pages that way.
A practical design is to expose ordinary paginated URLs, link them where appropriate, and enhance the browser experience with automatic loading. Keep the URL and content mapping stable so a crawler or user can request a specific chunk directly.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Best Value
Or skip the browser setup
If your goal is a clean image or PDF of a page rather than extracting every record, ScreenshotNeo provides a single HTTP request. It accepts the cookie or consent banner like a visitor, removes more than 60 known consent platforms plus newsletter popups and chat widgets before capture, and lets you turn each cleanup step off. Bot checks, CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed; response headers identify the page verdict and whether it was billed. Its MCP server exposes take_screenshot, get_page_info and capture_pdf to Claude, Cursor and other MCP clients.
See the ScreenshotNeo API documentation for all options, including full-page lazy-image loading, CSS-selector element capture, dark mode, device presets, retina scale, PDF paper and page controls, custom CSS and JavaScript, clicks, selector or network-idle waits, request blocking, headers, cookies, user agents, authorization, timezone, geolocation, transparency, resizing, TTL caching, signed links, asynchronous webhooks, bulk capture and usage data.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpimport requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);The Free plan includes 1,000 shots each month without a card. Paid plans start at $5 for 3,000 shots; yearly billing gives two months free. Create a free ScreenshotNeo account to try it.
ScreenshotNeo plans
| Plan | Allowance | Price |
|---|---|---|
| Free | 1,000 shots/month | $0, no card |
| Starter | 3,000 shots | $5 |
| Growth | 15,000 shots | $15 |
| Pro | 60,000 shots | $39 |
| Scale | 250,000 shots | $99 |
| Business | 1,000,000 shots | $249 |
All features are included on every plan. These capture plans do not replace Ruby automation when you must inspect, transform or persist each feed item; they are a simpler option for rendered-page evidence.
Quick decision guide
| Need | Best approach | Reason |
|---|---|---|
| Ruby browser tests | Watir | Ruby-oriented API with scrolling support. |
| Existing WebDriver suite | Selenium Ruby | Reuse direct driver control and test infrastructure. |
| Feed inside a panel | Scroll the panel | The window may not own the scroll position. |
| Rendered screenshot or PDF | ScreenshotNeo | Clean capture without maintaining browser setup; only clean shots are billed. |
Frequently Asked Questions
Can I use a fixed sleep after every scroll?
You can add a small delay for a site that needs it, but a sleep alone cannot prove that new content arrived. Wait for a count, identity, loading transition plus a result, or end marker, and keep a timeout.
How do I know whether the window or a panel scrolls?
Inspect the element with developer tools and look for an independently changing scroll position, an overflow setting, and a scroll height larger than its client height. Then observe item changes inside that element.
Is Playwright the recommended Ruby solution here?
The reviewed material documents useful Playwright patterns but does not establish the current status or availability of a Ruby binding. Watir and Selenium Ruby are the documented choices for this article.
What should a crawler receive from an infinite-scroll site?
Give every content chunk a stable, unique URL and make relevant content load when visible without requiring a user scroll or click. Browser automation and search crawlability are separate concerns.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
