Selenium is useful when the data you need appears only after a page runs JavaScript or requires browser interaction. A successful page navigation is not proof that the application has finished rendering: wait for the specific content or state your next step depends on, then extract it and close the browser session.
What Selenium does in a JavaScript scraper
Selenium WebDriver lets JavaScript code control a real browser through Selenium’s language binding and a browser driver. That makes it possible to inspect browser-rendered content and interact with the page, rather than only read the initial HTML response. Selenium can run a browser locally or remotely; its documentation points to Selenium Grid for remote execution and scaling. Selenium documentation
This capability has an operational cost: a browser requires more setup and resources than a direct HTTP request. Use it when browser rendering or interaction is needed, not merely because the target URL is a website.
When to use Selenium instead of a direct HTTP request
| Approach | Best fit | Trade-off |
|---|---|---|
| Direct HTTP request | The data is already available in the server response or through a documented data interface. | Usually simpler to implement and maintain, but it does not provide browser-rendered behavior or user-like interaction. |
| Selenium WebDriver | The required content is added or changed by client-side JavaScript, or the workflow needs browser interaction. | Offers browser control and rendered-page access, at the cost of browser runtime, resource use and added operational complexity. |
There is no universal speed or cost threshold established here; the choice depends on the target page and the data you actually need. Selenium’s page-load strategy can change whether navigation waits for all page assets, but a less restrictive strategy is safe only if your later steps wait for the application state they require. Selenium waiting strategies
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
#1 Best Overall
Install Selenium’s JavaScript binding
Use a supported Node.js runtime and check Selenium’s current JavaScript API page for its current minimum version, since runtime requirements can change. The page currently identifies Node.js 22 or newer and names the package selenium-webdriver. Selenium JavaScript API
- Install Node.js if it is not already available.
- Create a project directory and initialize it with
npm init -y. - Install Selenium’s JavaScript binding with
npm install selenium-webdriver. - Choose a browser and ensure its installation and driver setup are available for your environment. Consult Selenium’s current documentation for browser-specific setup; do not assume one driver setup applies to every browser or platform.
Run a small scraper and wait for the rendered result
This example visits a page with a result container whose contents are updated by JavaScript. Replace the URL and CSS selector with the page and element you are authorized to access. The explicit wait checks for visibility before reading the element’s text.
const { Builder, By, until } = require('selenium-webdriver');
async function scrape() {
const driver = await new Builder().forBrowser('chrome').build();
try {
await driver.get('https://example.com/results');
const results = await driver.wait(
until.elementLocated(By.css('.results')),
10000
);
await driver.wait(until.elementIsVisible(results), 10000);
const text = await results.getText();
console.log(text);
} finally {
await driver.quit();
}
}
scrape().catch((error) => {
console.error(error);
process.exitCode = 1;
});The sequence is deliberate: build the browser, navigate, locate the needed content, wait until it is usable, extract only what the task needs, and quit in finally so a failed wait or extraction does not leave the session running. The selector and timeout are examples; choose a condition and timeout that match the target page and your application’s needs.
Wait for the condition your next step needs
WebDriver navigation waits according to the configured page-load strategy and a document ready state. That state concerns assets declared in the HTML; scripts can continue changing the page afterward. A result element may therefore be absent or hidden when the next WebDriver command runs. Selenium waiting strategies
Prefer an explicit, condition-based wait. Selenium’s JavaScript API offers conditions such as element location and visibility; the right one depends on the next action. For example, wait for an element to exist before reading its attributes, or for it to become visible before interacting with it. Selenium JavaScript API
- Wait for presence when the element must exist in the DOM.
- Wait for visibility when the element must be visible before reading or interacting with it.
- Wait for a meaningful application state when presence alone is insufficient, such as a result list receiving content or a loading indicator disappearing. Select a condition that accurately represents readiness for your next action.
A fixed sleep is a poor default: it may be too short when a page is slow and wastes time when it is fast. Selenium also warns against mixing implicit and explicit waits in the same session because their combined timing can be unpredictable. Use a consistent explicit-wait strategy rather than adding an implicit wait and hoping the durations compose cleanly. Selenium waiting strategies
Extract only what you need
Once the required state is ready, use a locator for the smallest relevant element and read the needed text or attributes. Narrow extraction makes the script easier to inspect and reduces dependence on unrelated page markup. Selectors should match the page’s current DOM; a locator that worked before a site redesign may no longer identify the same content.
For a list of repeated results, locate the result elements and read the relevant field from each rather than capturing the whole page as an undifferentiated string. Keep the browser cleanup in a finally block when adding more extraction steps, so errors do not skip session shutdown.
Free tools Windows power users keep installed
One-click scans. No signup required.
Troubleshoot common failures
Element not found or wait times out
- Check whether the selector still matches the current DOM and whether the content is inside a frame or another context your script has not entered.
- Confirm that the page actually reached the state you are waiting for. Navigation completing does not establish that client-side content has appeared.
- Wait for the specific missing precondition, such as the result container appearing or becoming visible. Do not only increase the timeout without identifying which state is missing.
Element exists but is not usable
Presence in the DOM is not the same as visibility or readiness for interaction. Use a visibility wait or another condition that corresponds to the operation you plan to perform.
Script finishes before results appear
Inspect the next command after navigation and identify what it assumes. Add an explicit wait for that exact element or application state before reading or clicking. A longer fixed sleep may hide the race without reliably fixing it.
Wait duration behaves unpredictably
Check whether the session uses both implicit and explicit waits. Selenium warns that combining them can lead to unpredictable total wait times; use explicit waits for the conditions that matter instead.
Browser or driver does not start
Check that your Node.js runtime meets the current Selenium JavaScript API requirement, the package is installed, and the selected browser is available with the corresponding driver setup for your environment. Selenium’s setup details can change, so consult its current documentation rather than applying an outdated browser-specific instruction. Selenium JavaScript API
Reliability, runtime and scaling considerations
A real browser can provide the rendering and interaction your extraction depends on, but it adds resource use, session management and more failure points than requesting data directly. Limit browser work to pages that need it, extract the smallest useful amount, and close sessions even when a step fails. The selected Selenium documentation establishes browser-control and wait behavior, but does not provide a benchmark comparing Selenium throughput with HTTP-only scraping.
If local browser execution becomes an operational constraint, Selenium documents remote execution and Selenium Grid as options to explore. The appropriate setup depends on your workload and infrastructure; the documentation does not establish a particular provider’s price or capabilities. Selenium documentation
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Access responsibly
Check the target site’s terms, permissions and obligations that apply to your use before collecting data. MDN describes robots.txt as a publicly accessible file at a site’s root that gives crawler instructions. It is optional, some robots ignore it, and it is not a security mechanism or proof of permission. MDN: robots.txt
Or skip the browser setup
If your goal is a screenshot rather than structured extraction, ScreenshotNeo provides a website screenshot API and MCP server. A single GET request can return a PNG, JPEG, WebP or PDF. It is not a replacement for Selenium when you need to parse data or interact with a page as part of a scraping workflow.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsBest Value
For a screenshot of a page, the cURL call is:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo API documentation for options and response details. Cookie banners are accepted like a visitor and more than 60 known consent platforms, newsletter popups and chat widgets can be removed before capture; each step can be turned off. Bot checks, blank pages, timeouts, failed loads and cache hits cost nothing, with response headers identifying the page verdict and billing status. Its MCP server provides take_screenshot, get_page_info and capture_pdf tools for AI agents and MCP clients. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000 shots.
Sign up for ScreenshotNeo’s free plan to get 1,000 screenshots a month with no card.
Frequently Asked Questions
Can Selenium scrape content that is added after the page loads?
Yes. Navigate to the page, then wait for the relevant rendered element or state before extracting it.
Does robots.txt give permission to scrape a site?
No. It provides crawler instructions, not access control or legal permission.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
