What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Use an explicit wait tied to a real result condition, scroll the element that actually owns the results, and stop after bounded checks with no DOM growth. A completed document.readyState only tells you that the initial document finished loading; a JavaScript application can still be rendering search cards.
First, confirm that your automation is allowed
LinkedIn’s current Help guidance says third-party software or browser extensions that scrape, modify, or automate activity are not allowed and may violate its User Agreement or privacy legislation. The User Agreement effective November 3, 2025 prohibits scripts or robots that scrape or copy the Services, bypass access controls or use limits, and other unauthorized automated methods. LinkedIn’s crawling terms state: “Automated Crawling & Indexing without the express permission of LinkedIn is strictly prohibited.”
Use the Selenium pattern below only for a page you own, an authorized test environment, or an activity covered by written permission from LinkedIn. For legitimate data access, prefer an approved API, export, or partner interface. Do not use this method to evade a login, CAPTCHA, rate limit, or other access control.
Why a LinkedIn search page appears to stop early
Lazy loading defers noncritical or non-visible content until it is needed. Infinite scroll is one form of lazy loading: the page requests more items when the user approaches the end of the current list. Selenium can return control while the initial HTML is complete, before the application has rendered the next batch.
#1 Best Overall
Two details cause most failures:
- Ready state is not application readiness. A single-page application may report
completewhile JavaScript is still fetching and inserting cards. - The viewport may not be the scroll target. Search results can sit inside an element with its own scrollbar. Scrolling
windowthen moves the wrong surface and never triggers loading.
The reliable loading algorithm
- Navigate to the authorized search URL.
- Wait for the first result-card condition with an explicit Selenium wait.
- Inspect the page and identify the results container and a stable card locator. Treat selectors shown below as placeholders, not guaranteed LinkedIn markup.
- Record progress, such as card count and the container’s
scrollHeight. - Scroll the results container (or the viewport when that is the tested behavior) close to its bottom.
- Wait for measurable progress: a larger card count, a loading indicator disappearing, or an increased scroll height.
- Repeat with hard limits such as a maximum card count, maximum iterations, and a no-growth counter. Stop when the page reports no more results or repeated checks show no growth.
- Deduplicate using a stable identifier available in the authorized DOM, and retain only fields covered by your authorization.
The important synchronization primitive is the condition, not a fixed sleep. A short delay can be useful for diagnostics, but it is brittle when network and rendering times vary.
Complete Python Selenium example
This example uses a viewport scroll and a result-card count as its progress signal. Replace the URL and selectors after inspecting your authorized test page.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
AUTHORIZED_URL = 'https://www.linkedin.com/search/results/people/?keywords=AUTHORIZED_TERM'
CARD = (By.CSS_SELECTOR, '[data-result-card]') # discover on your authorized page
options = webdriver.ChromeOptions()
# options.add_argument('--headless=new') # enable only when your test supports it
driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 15)
try:
driver.get(AUTHORIZED_URL)
wait.until(EC.presence_of_element_located(CARD))
max_cards = 100
max_iterations = 20
stalled = 0
previous_count = 0
for _ in range(max_iterations):
cards = driver.find_elements(*CARD)
current_count = len(cards)
if current_count >= max_cards:
break
driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')
try:
wait.until(lambda d: len(d.find_elements(*CARD)) > current_count)
stalled = 0
except TimeoutException:
stalled += 1
if stalled >= 2:
break
previous_count = current_count
cards = driver.find_elements(*CARD)
for card in cards:
# Extract only data your authorization permits.
print(card.text)
finally:
driver.quit()The values 15 seconds, 100 cards, 20 iterations, and two stalled checks are conservative example bounds, not a performance guarantee. Adjust them to the authorized page and your test objective.
When results are inside an inner scrolling panel
Find the element whose scrollbar moves when you drag the results pane. Replace the viewport scroll and measure that element instead:
Rank #2
SCROLLER = (By.CSS_SELECTOR, '[data-results-scroll-container]') # placeholder
scroller = wait.until(EC.presence_of_element_located(SCROLLER))
for _ in range(max_iterations):
cards = driver.find_elements(*CARD)
current_count = len(cards)
old_height = driver.execute_script('return arguments[0].scrollHeight;', scroller)
driver.execute_script(
'arguments[0].scrollTop = arguments[0].scrollHeight;', scroller
)
try:
wait.until(lambda d: (
len(d.find_elements(*CARD)) > current_count or
d.execute_script('return arguments[0].scrollHeight;', scroller) > old_height
))
stalled = 0
except TimeoutException:
stalled += 1
if stalled >= 2:
breakIf the page virtualizes its list, old nodes can be recycled rather than retained. In that case, extract each card as it appears and deduplicate by a stable ID or canonical profile URL instead of assuming that the final DOM contains every item.
Choosing a completion signal
| Signal | What it proves | Typical limitation |
|---|---|---|
| Card count increases | New result nodes were inserted | Virtualized lists may recycle nodes; count can stay constant |
| Loading indicator disappears | One fetch/render phase ended | The indicator may be absent, reused, or hidden before cards appear |
| Scroll height increases | The scroll surface grew | Layout changes can alter height without new results |
| “No more results” state | The application explicitly ended pagination | Requires a stable, authorized locator for that state |
Combining card count with scroll height, then enforcing a no-growth stop rule, is safer than trusting any one signal. Keep a maximum result count and iteration limit so a broken page cannot create an unbounded loop.
Deduplication and data minimization
Choose an identifier that remains stable across rerenders, such as an authorized data attribute or canonical URL. Do not use display text alone when names can repeat. Store only the fields your test or integration requires, define a retention period, and avoid collecting contact details or other personal data that is outside the written authorization.
Common failures and fixes
The first-card wait times out
The browser may be on a sign-in or consent screen, the search is not authorized, the selector is stale, or the page returned an error. Capture the current URL and a diagnostic screenshot in your test environment, verify the account and permission, and rediscover the card locator. Do not attempt to bypass a challenge.
Scrolling changes nothing
You may be scrolling window while an inner panel owns the scroll. Inspect which element’s scrollTop changes in the browser’s authorized test session, then scroll that element and measure its scrollHeight.
Ready state is complete but cards are missing
Replace the ready-state check with an explicit wait for the first card and a progress condition after each scroll. JavaScript rendering commonly continues after the browser reports the document complete.
The loop exits after one iteration
Check whether the card locator already matches a hidden template, whether the list is virtualized, or whether the application intentionally returned no more results. Log card count, scroll height, and the presence of a loading or end-state element on every iteration.
Cards become stale during extraction
Infinite-scroll frameworks can replace nodes while you read them. Extract immediately after each progress event, or reacquire the card collection before each read. Save the stable identifier first, then fetch optional fields only while the element is attached.
A bot check or CAPTCHA appears
Stop the run. A challenge is an access control, not a loading condition. Use an approved LinkedIn integration or request written permission rather than trying to defeat it.
Performance, reliability, and cost considerations
- Performance: Waiting on DOM progress avoids unnecessary long sleeps, while a maximum-card limit prevents collecting more than the test needs. Browser startup, authentication, network latency, and rendering remain the dominant costs.
- Reliability: Record the URL, selector version, counts, heights, and stop reason. Selector changes are an expected maintenance issue because LinkedIn does not guarantee search markup stability.
- Resource use: Reuse a WebDriver session for a bounded test suite, but clear state between independent authorization contexts. Close the driver in a
finallyblock. - Cost: Selenium itself has no per-result API charge, but it consumes your browser, compute, proxy, and maintenance budget. An approved API or export may be cheaper and more stable for recurring data access.
Or skip the browser setup
If your goal is a visual capture of an authorized page rather than extracting search data, ScreenshotNeo returns a screenshot or PDF through one request. It is not a substitute for permission to access LinkedIn data, but it removes the need to maintain a local browser for a visual artifact.
Use the API parameters documented in the ScreenshotNeo documentation:
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://www.linkedin.com/search/results/people/?keywords=AUTHORIZED_TERM -o shot.webpimport requests
r = requests.get(
'https://api.screenshotneo.com/v1/shot',
params={
'access_key': 'YOUR_API_KEY',
'url': 'https://www.linkedin.com/search/results/people/?keywords=AUTHORIZED_TERM'
},
timeout=90,
)
r.raise_for_status()
open('shot.webp', 'wb').write(r.content)const q = new URLSearchParams({
access_key: 'YOUR_API_KEY',
url: 'https://www.linkedin.com/search/results/people/?keywords=AUTHORIZED_TERM'
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const fs = await import('node:fs/promises');
await fs.writeFile('shot.webp', Buffer.from(await res.arrayBuffer()));ScreenshotNeo accepts options for full-page captures with lazy images loaded, CSS-element capture, dark mode, device and viewport settings, retina scale, PDF paper and page ranges, custom CSS or JavaScript, clicks, selector waits, delays, network-idle waits, request or resource blocking, headers, cookies, user agents, authorization, timezone, geolocation, transparent backgrounds, resizing, chosen cache TTLs, signed image links, asynchronous webhooks, bulk capture of up to 100 URLs per call, usage reporting, and an OpenAPI specification. Parameter names used by other screenshot APIs also work.
Before a capture, its cleaner accepts cookie or consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be disabled. Only clean shots are billed. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and the response identifies the outcome with X-Page-Verdict and X-Billed headers. Its MCP server exposes take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients.
The Free plan includes 1,000 shots per month with no card. Paid plans are Starter $5 for 3,000 shots, Growth $15 for 15,000, Pro $39 for 60,000, Scale $99 for 250,000, and Business $249 for 1,000,000; yearly billing gives two months free, and every feature is included on every plan. Create a free ScreenshotNeo account to try the 1,000-shot allowance without adding a card.
Frequently Asked Questions
Can I rely on the number of visible cards as the total result count?
No. A virtualized list can recycle a fixed set of nodes, and search pagination can be capped or personalized. Treat the count as a progress signal, not as an authoritative total.
Recommended Free Tools
Should I add a longer sleep when the page is slow?
Prefer increasing the explicit wait timeout for the authorized environment and retaining a measurable condition. A fixed sleep does not tell the test whether rendering succeeded and makes fast runs unnecessarily slow.
Is a headless browser automatically safe for LinkedIn automation?
No. Headless mode changes how the browser is displayed, not whether the activity is authorized. The same LinkedIn restrictions and data-minimization requirements apply.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
