Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsTo extract values reliably, first determine whether the control is a native HTML <select> or a JavaScript widget made from elements such as <div>, <button>, or <li>. For a native control, wait until the required options have been populated, wrap it with Selenium’s Select helper, and read each option’s value attribute. For a custom widget, open it, wait for its rendered option elements, and read the site-specific attribute or visible text.
1. Install Selenium and create a driver
Install Selenium in the environment that will run the scraper or test:
python -m pip install -U seleniumThe examples below target Selenium 4’s Python API. Create a WebDriver for the browser you have installed. Selenium Manager can generally obtain a compatible driver automatically:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
try:
driver.get("https://example.com")
# extraction code goes here
finally:
driver.quit()Replace the URL and locators with those from the page you are automating. A browser reaching its configured page-load state does not mean that JavaScript has finished adding options.
#1 Best Overall
2. Extract every value from a native select
A native dropdown has a <select> element containing <option> elements. Selenium’s Select helper is designed specifically for that markup.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select, WebDriverWait
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
try:
driver.get("https://example.com/form")
wait = WebDriverWait(driver, 10)
select_element = wait.until(
lambda d: d.find_element(By.ID, "country")
)
country_select = Select(select_element)
values = [
option.get_attribute("value")
if option.get_attribute("value") is not None
else option.text
for option in country_select.options
]
print(values)
finally:
driver.quit()Select.options returns all options in document order. The code deliberately distinguishes a missing value attribute from an explicitly empty value="". Under HTML semantics, an option with no value attribute uses its text as the value; an explicitly empty value remains an empty string.
Values versus labels
The submitted value and the text a user sees are separate fields:
records = [
{
"value": option.get_attribute("value"),
"label": option.text.strip(),
}
for option in country_select.options
]
for record in records:
print(record["value"], record["label"])If your output must follow HTML fallback semantics, use the value-or-text expression shown earlier. If you need to preserve whether the attribute was absent, inspect get_attribute("value") directly and keep None distinct from "".
Removing a placeholder option
Many forms begin with a prompt such as “Choose a country” and an empty value. Filter it only when your application defines it as a placeholder:
real_values = [
option.get_attribute("value")
if option.get_attribute("value") is not None
else option.text
for option in country_select.options
if (option.get_attribute("value") or option.text.strip())
]Do not apply this filter blindly: an empty value can be an intentional choice.
3. Wait for JavaScript to populate the options
Finding the <select> is not the same as finding a populated list. Use an explicit wait for the state your extraction needs rather than a fixed sleep.
Wait for a known option value
from selenium.webdriver.support.ui import Select, WebDriverWait
from selenium.webdriver.common.by import By
wait = WebDriverWait(driver, 10)
def has_california(d):
select = Select(d.find_element(By.ID, "country"))
return any(
option.get_attribute("value") == "CA"
for option in select.options
)
wait.until(has_california)
select = Select(driver.find_element(By.ID, "country"))
values = [
option.get_attribute("value")
if option.get_attribute("value") is not None
else option.text
for option in select.options
]Wait for more than a placeholder
def has_real_options(d):
select = Select(d.find_element(By.ID, "country"))
return len(select.options) > 1
WebDriverWait(driver, 10).until(has_real_options)Choose a condition that represents readiness on the target site: a known value, a minimum count, a loading indicator disappearing, or a specific enabled state. Selenium’s documented default polling interval for WebDriverWait is 0.5 seconds. A fixed delay can waste time on fast responses and still fail when a slow response takes longer.
Free tools Windows power users keep installed
One-click scans. No signup required.
4. Extract only the selected option
Use selected-option properties when the question is “what is selected now?” rather than “what choices exist?”
select = Select(driver.find_element(By.ID, "country"))
first = select.first_selected_option
print(first.get_attribute("value"), first.text)
selected = [
{
"value": option.get_attribute("value"),
"label": option.text.strip(),
}
for option in select.all_selected_options
]A single-select normally has one selected option. A multi-select can return several through all_selected_options.
5. Handle dependent dropdowns
For country-and-state or category-and-product controls, changing the parent often causes the child element to be replaced or repopulated. Reacquire the child and wait again after every parent change.
from selenium.webdriver.support.ui import Select, WebDriverWait
from selenium.webdriver.common.by import By
wait = WebDriverWait(driver, 10)
parent = Select(driver.find_element(By.ID, "country"))
parent.select_by_value("US")
def state_list_ready(d):
child = Select(d.find_element(By.ID, "state"))
return any(option.get_attribute("value") == "CA"
for option in child.options)
wait.until(state_list_ready)
# Reacquire after the page may have replaced the element.
child = Select(driver.find_element(By.ID, "state"))
states = [
option.get_attribute("value")
if option.get_attribute("value") is not None
else option.text
for option in child.options
]Do not retain a stale WebElement reference across an update that rebuilds the child control. If the application keeps the element but changes its contents, reacquiring it is still a simple way to avoid reading an old state.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →6. Custom JavaScript dropdowns
If the live DOM contains no native <select>, do not pass the control to Select. Custom widgets commonly use a button that opens a listbox, with options represented by div or li elements. Their selectors and value attributes are application-specific.
- Open browser developer tools and inspect the rendered DOM after the page and widget have loaded.
- Identify a stable trigger: an accessible label, stable ID, or semantic role.
- Click the trigger and wait until the option container is present or visible.
- Read the option’s stable attribute, such as a site-defined
data-value, and its visible text separately.
This illustrative pattern uses ARIA roles where the page provides them; adapt the locators to the actual markup:
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
wait = WebDriverWait(driver, 10)
trigger = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "[aria-label='Country']"))
)
trigger.click()
options = wait.until(
EC.visibility_of_all_elements_located(
(By.CSS_SELECTOR, "[role='option']")
)
)
items = [
{
"value": item.get_attribute("data-value"),
"label": item.text.strip(),
}
for item in options
]Some widgets render their menu in a portal elsewhere in the document, virtualize rows while scrolling, or load choices only after a search keystroke. In those cases, inspect the post-click DOM and wait for the specific state that proves the complete list is available. Generated class names and positional XPath are fragile because libraries can change them without changing the user interface.
7. Selecting by value, text, or index
For native controls, Selenium supports several selection methods:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
select.select_by_value("CA")
select.select_by_visible_text("California")
select.select_by_index(2)Prefer a stable value or exact visible text. Indexes depend on ordering and break when a placeholder, localization, or newly inserted option changes the list. For custom widgets, click the matching rendered option yourself and wait for the widget’s selected state.
8. Troubleshooting
“Select only works on <select> elements”
The control is custom markup, not a native select. Inspect the DOM, then use the trigger and option locators described above.
The list is empty
The element may exist before asynchronous data arrives. Wait for a known option or a meaningful count. Also check that the parent selection, authentication, request headers, and network calls required by the page are valid.
TimeoutException
Confirm the locator, URL, frame, and expected value. If the dropdown is inside an iframe, switch into it before locating the control:
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11frame = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, "iframe"))
)
driver.switch_to.frame(frame)Switch back with driver.switch_to.default_content() when finished. Increase the timeout only after verifying that the condition is correct; a longer wait does not fix a wrong selector.
StaleElementReferenceException
The framework replaced the element after a parent change or network response. Locate the element again inside the wait or immediately before extraction.
Duplicate labels
Visible text may not be unique. Extract and use the underlying value or another stable attribute. If values are also duplicated, retain the option’s order and any identifying metadata your application exposes.
Hidden or virtualized options
A widget may render only visible rows. Scroll the list using the widget’s supported interaction, wait for additional rows, and deduplicate values. Do not assume that elements absent from the DOM are merely hidden.
Headless and headed results differ
Compare viewport size, user agent, locale, timezone, authentication, and responsive breakpoints. Set a deterministic window size and capture diagnostic HTML or screenshots when a failure occurs.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.9. Reliability, performance, and data quality
- Wait on application state, not arbitrary time.
- Use one driver session for related page transitions, but always quit it in a
finallyblock. - Reacquire controls that may be replaced.
- Keep raw values and labels so downstream code can distinguish display text from submitted data.
- Normalize whitespace only for presentation; do not silently alter identifiers.
- Log the URL, locator, option count, and readiness condition when extraction fails.
- Respect authentication, robots policies, rate limits, and the site’s terms.
For large jobs, avoid launching a new browser for every URL. Reuse a controlled session where isolation permits, and process independent pages with a bounded number of workers. Excessive parallel browsers increase memory use and can trigger site defenses.
10. Or skip the browser setup
If you need a rendered page image rather than DOM values, ScreenshotNeo provides a GET endpoint that returns PNG, JPEG, WebP, or PDF. Its cleanup steps accept cookie or consent banners and remove more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be disabled. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing status. It also offers an MCP server for AI clients with take_screenshot, get_page_info, and capture_pdf.
See the ScreenshotNeo documentation for parameters and response details. A one-call cURL example:
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);The Free plan includes 1,000 screenshots per month with no card. Paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account to try it.
Frequently Asked Questions
Can Selenium read options before they are visible?
It can read elements present in the DOM, but a dynamic application may not have inserted the final options yet. Wait for the option value, count, or other state your extraction requires.
How do I know whether a dropdown is native?
Inspect the live DOM. A native control is a select element containing option elements; buttons, divs, list items, or listboxes indicate a custom widget that needs page-specific interaction.
Should I save option text or value?
Save both when possible. Text is the user-facing label; value is the submitted identifier, and they can differ.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
