Recommended Free Tools
Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
To scrape a value with Selenium, open the page, locate the element that contains the value, wait until it is ready if the page is dynamic, and read the right representation: use rendered text for visible copy, or an attribute or runtime property for values such as form inputs. The example below uses Python and an explicit wait; it also shows how to collect multiple matching values and handle a missing result.
What Selenium can read from a page
Selenium WebDriver controls a real browser and lets your script find page elements and inspect their information. It is useful when the value you need is rendered in a website’s interface or populated by page JavaScript. The value may be visible text, text in the DOM, an HTML attribute, or a current runtime property. Those are not interchangeable: choose the representation that holds the data you want.
This is browser automation, not a guarantee that every page can be accessed or that a site permits automated collection. Before collecting data, check the site’s terms and applicable rules, and avoid collecting personal or sensitive information without an appropriate basis.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Set up Python, a browser, and Selenium
A basic Selenium run needs the Selenium language binding, a browser, and a compatible browser driver. The official Selenium getting-started guide covers setup and the first WebDriver script: Selenium getting started. The example here assumes Python, Chrome, and a working Chrome WebDriver setup; if you use another browser, use its corresponding WebDriver.
#1 Best Overall
Install the Python binding in the environment where the script will run:
python -m pip install seleniumSave the following as scrape_values.py. Replace the sample URL and CSS selectors with ones from the page you are authorized to inspect.
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
URL = "https://example.com/products"
options = webdriver.ChromeOptions()
# Uncomment for a headless run, such as on a server:
# options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 15)
try:
driver.get(URL)
# Wait for the specific product value to exist before reading it.
price = wait.until(
EC.visibility_of_element_located((By.CSS_SELECTOR, ".product-price"))
)
print("First visible price:", price.text.strip())
# Collect all currently matching product names.
names = driver.find_elements(By.CSS_SELECTOR, ".product-name")
if not names:
print("No product names matched the selector")
else:
for name in names:
value = name.text.strip()
if value:
print(value)
# A form input's current value is a runtime property.
search_box = driver.find_element(By.CSS_SELECTOR, "input[name='q']")
print("Current search input:", search_box.get_property("value"))
except TimeoutException:
print("Timed out waiting for the product price to become visible")
finally:
driver.quit()The script waits up to 15 seconds for the price element to become visible, reads its rendered text, then uses a plural finder for the product-name matches. Selenium’s plural find methods return a collection and return an empty list when there are no matches, so the empty-list branch is intentional. The input example reads the current value property rather than assuming the original HTML attribute contains the value currently shown in the field.
For the official element-finding and element-information APIs, see Finding web elements and Information about web elements.
Choose the right locator and value representation
Find one element or many
Use find_element when the page should contain one target, such as a price for a specific product. It returns one matching element and raises an exception if it cannot find a match. Use find_elements when collecting a repeated set, such as product cards, search results, or table rows. It returns all matches found at that moment; an empty collection means the locator did not match any elements at that moment.
Prefer selectors that express the element’s role or stable identity, such as a meaningful name, an ID intended for that control, or a distinctive class. Avoid selecting an element only because it is currently the third div on the page: markup changes can make positional selectors point somewhere else. For a repeated record, locate the record containers first and then inspect the intended child within each one, rather than combining unrelated page text into a single result.
Rendered text, DOM text, and values
- Visible rendered text: use
element.textwhen you want the text presented by the element as rendered in the browser. Strip surrounding whitespace if it is not meaningful to your result. - DOM text content: use
element.get_property("textContent")if the DOM text itself is the target, including text that may not be presented as visible rendered text. This can include hidden or otherwise non-visible text, so confirm that this is what you intend to collect. - HTML attribute: use
element.get_dom_attribute("name")when you need the attribute in the page markup, such as a link’shrefor an input’s initialvalueattribute. - Current runtime property: use
element.get_property("value")for the live value of an input after a user or script has changed it. A DOM attribute may not reflect that current state.
Selenium distinguishes rendered text from DOM text and from retrieving attribute or property values. For that reason, validate a sample against what the page actually displays and what your downstream task needs, rather than treating every visible-looking value as plain text.
Wait for JavaScript-driven values
A browser navigation completing at its configured page-load readiness state does not prove that the page’s later JavaScript has finished rendering or updating the target value. A script that reads immediately after driver.get() can therefore race the page: the selector may not exist yet, a placeholder may still be present, or the value may change after it is read.
The example uses an explicit wait for visibility of the target price. Other useful conditions include waiting for presence when visibility is not required, or waiting for a particular text/value condition when the element exists before its data is populated. Match the condition to the question your scraper needs answered: “element exists” is not the same as “correct value is present.” Selenium’s wait guide describes synchronization options and warns, “Do not mix implicit and explicit waits.” See Selenium waiting strategies.
Do not solve a slow page by adding an arbitrary long sleep as the normal synchronization method. A fixed delay can waste time on fast runs and still be too short on slow ones. A condition-based explicit wait stops when the condition succeeds or its timeout is reached.
Collect repeated values as structured records
When values belong together, such as a product name and price, iterate through each record container and read its children. This avoids accidentally pairing the first name on a page with the first price from a different section.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutecards = driver.find_elements(By.CSS_SELECTOR, ".product-card")
products = []
for card in cards:
name = card.find_element(By.CSS_SELECTOR, ".product-name").text.strip()
price = card.find_element(By.CSS_SELECTOR, ".product-price").text.strip()
products.append({"name": name, "price": price})
for product in products:
print(product)Adapt selectors and missing-element handling to the page. A card without a price can raise an exception inside the loop; decide whether that record should be skipped, retained with a missing field, or treated as a page-layout error. If the page replaces cards while you iterate, previously located element references may no longer point to attached elements. In that case, wait for the update to finish and locate the current records again.
Common errors and how to fix them
- No such element: the selector may be wrong, the element may not have loaded yet, or it may be inside a different browsing context. Inspect the page’s current DOM, correct the selector, and wait for the relevant condition before looking it up.
- Timeout waiting for a condition: confirm the page opened the expected URL and that the target can satisfy the chosen condition. An element that exists but remains hidden will not satisfy a visibility wait. Use a condition that matches your need, or investigate whether the page is showing an error, consent prompt, or different state.
- Empty results from
find_elements: this is a valid empty collection, not a Selenium failure. Check the selector and timing, and make sure the elements are in the current page context. - Blank text despite a visible value: the value may be in an input property, a child element, or updated asynchronously. Inspect the target element and use the relevant property or child locator; wait for the value itself if JavaScript populates it after page load.
- Stale element reference: the page may have re-rendered after the element was located. Wait for the update and find the element again rather than relying on an old reference.
- Driver or browser startup failure: check that the browser is installed and that the driver setup is compatible with the browser and execution environment. Selenium’s setup guide covers the required components: Getting started.
Performance, reliability, and scaling
For a small one-off extraction, a local WebDriver session is usually the simplest shape: start one browser, navigate, wait only for the conditions needed, collect values, and quit in a finally block. Browser work is heavier than reading already-available structured data, so if a site provides an authorized API or downloadable data source that meets your needs, compare that route before automating the interface.
For larger runs, reliability depends on stable selectors, explicit synchronization, handling empty or changing results, and closing sessions even when extraction fails. Selenium documents Grid as a route for scaling execution across infrastructure; it is an optional deployment choice, not a prerequisite for a local script. See Selenium getting started for the project’s setup and Grid direction.
Make output auditable: retain the page URL and a timestamp alongside extracted values when appropriate, and distinguish a genuinely empty field from a failed page or selector mismatch. Do not silently convert errors to empty strings; doing so can make incomplete extraction look like valid data.
Or skip the browser setup
If what you need is a visual record of a page rather than structured values from its elements, a screenshot API can avoid installing and running a browser in your own script. ScreenshotNeo returns a screenshot or PDF from one GET request; it is not a substitute for Selenium when you need to extract a specific text node, input property, or record as structured data.
Best Value
For example, this cURL request saves a WebP screenshot of the target page. See the ScreenshotNeo API documentation for request options.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpScreenshotNeo accepts cookie or consent banners as a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; those steps can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and the response identifies the page verdict and billing status in headers. Its MCP server gives AI agents tools for screenshots, page information, and PDF capture. The Free plan includes 1,000 shots a month without a card; paid plans start at $5 for 3,000 shots.
Sign up for ScreenshotNeo’s free plan to try 1,000 screenshots a month with no card.
Free tools Windows power users keep installed
One-click scans. No signup required.
Frequently Asked Questions
Can Selenium read a value that is not visible on the page?
Yes, if the value is present in the page’s DOM or element properties. Choose the relevant representation, such as textContent or a runtime property, and confirm that it is the intended data.
Does Selenium automatically know which page value I want?
No. Your locator and retrieval method determine which element and representation your script reads.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
