Free tools Windows power users keep installed
One-click scans. No signup required.
To select a value after a known BeautifulSoup node, first identify the relationship in the parsed tree. If both elements share a parent, use find_next_sibling() (or find_next_siblings() for several results). If the target is merely later in document order, use find_next() or a carefully bounded next_elements loop. Then call get_text(strip=True) or process stripped_strings to obtain clean text.
Start with the tree relationship
BeautifulSoup does not treat “the value between two nodes” as one universal operation. HTML relationships determine the correct method:
- Sibling: both tags have the same parent and are at the same level.
- Next parse-tree item: the literal item after a tag may be whitespace or punctuation, not another tag.
- Later in document order: the target can be nested inside another element or located in a later section.
Beautiful Soup’s documentation notes that in real documents, .next_sibling or .previous_sibling will usually be a string containing whitespace. That is why a sibling search is generally safer than assuming node.next_sibling is the desired element.
Select the next matching sibling
For a label/value structure such as a definition list, find the anchor tag and ask for the next sibling with the required name:
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errors#1 Best Overall
from bs4 import BeautifulSoup
html = """
<dl>
<dt>Price</dt>
<dd>19.99</dd>
</dl>
"""
soup = BeautifulSoup(html, "html.parser")
label = soup.find("dt", string="Price")
value_node = label.find_next_sibling("dd") if label else None
value = value_node.get_text(strip=True) if value_node else None
print(value) # 19.99find_next_sibling("dd") skips intervening whitespace and returns the first later dd sibling. The conditional check prevents an AttributeError when the label is absent. Returning None for a missing value also makes the failure explicit instead of silently producing an empty string.
Match a label robustly
Exact string="Price" works only when the dt contains one text node with that exact value. For nested markup or extra whitespace, use a predicate or normalize the text:
label = soup.find(
"dt",
string=lambda text: text and text.strip().casefold() == "price"
)
value_node = label.find_next_sibling("dd") if label else None
value = value_node.get_text(" ", strip=True) if value_node else NoneIf a label contains nested tags, search the candidate elements and compare get_text():
label = next(
(tag for tag in soup.find_all("dt")
if tag.get_text(" ", strip=True).casefold() == "price"),
None
)Get all later siblings
Use find_next_siblings() when a node is followed by multiple values at the same level. It returns a list rather than one tag:
html = """
<ul>
<li class="label">Colors</li>
<li>red</li>
<li>green</li>
<li>blue</li>
</ul>
"""
soup = BeautifulSoup(html, "html.parser")
label = soup.select_one("li.label")
values = [
item.get_text(" ", strip=True)
for item in label.find_next_siblings("li")
] if label else []
print(values) # ['red', 'green', 'blue']You can constrain the result with attributes, a CSS class, or a custom filter. Remember that this method keeps searching at the same parent level; it does not descend into each sibling’s children to find unrelated matches.
When the target is not a sibling
If the second node is nested or simply appears later elsewhere in the document, use document-order traversal.
find_next() for the first later match
heading = soup.find("h2", string="Specifications")
value_node = heading.find_next("span", class_="value") if heading else None
value = value_node.get_text(" ", strip=True) if value_node else Nonefind_next() starts after the anchor and can cross nesting boundaries. That flexibility also creates a risk: it may return a matching element from a later, unrelated section. Scope the search to a known container whenever possible:
card = soup.select_one("article.product-card")
label = card.find("span", class_="price-label") if card else None
value = label.find_next("span", class_="price") if label else NoneIterate with next_elements when you need a stopping rule
next_elements walks subsequent tags and strings in parse order, including descendants. This is useful when the value can have variable markup, but you should stop at a known boundary:
section = soup.select_one("section.details")
result = None
if section:
for item in section.next_elements:
if getattr(item, "name", None) == "h2" and item is not section:
break
if getattr(item, "name", None) == "span" and "price" in item.get("class", []):
result = item.get_text(" ", strip=True)
breakWithout a scope or boundary, a document-order loop can accidentally capture a later value with the same class.
Inspect the literal next tree item
Use .next_sibling only when you need to inspect exactly what follows at the same level:
node = soup.find("dt")
item = node.next_sibling if node else None
print(type(item).__name__, repr(item))The result may be a NavigableString containing a newline, spaces, or punctuation. To reach the next tag manually, continue through siblings and test the tag name:
item = node.next_sibling if node else None
while item is not None and getattr(item, "name", None) != "dd":
item = item.next_sibling
value = item.get_text(" ", strip=True) if item else NoneIn most extraction code, find_next_sibling("dd") expresses this intent more clearly.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Extract text without joining unrelated content
get_text()
Call get_text(strip=True) for compact text. Pass a separator when descendants should remain visually separated:
text = value_node.get_text(" ", strip=True)For example, a node containing “$19” and a nested “.99” becomes “$19 .99” with a space separator, rather than concatenating words accidentally. Choose a separator appropriate to the markup.
stripped_strings
When each text fragment needs individual handling, iterate over stripped_strings:
parts = list(value_node.stripped_strings) if value_node else []
value = " ".join(parts) if parts else NoneSelect the narrowest target before extraction. Calling get_text() on a whole card or page can include labels, buttons, hidden notices, and unrelated content.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
CSS selectors for structural relationships
select_one() and select() are often clearer when the relationship is expressed by structure rather than relative movement:
value_node = soup.select_one("dl dt + dd")
value = value_node.get_text(" ", strip=True) if value_node else NoneThe adjacent-sibling selector + requires the dd to be immediately after the dt element (ignoring whitespace text nodes). For any later sibling, use ~:
later_values = soup.select("dl dt ~ dd")CSS selectors can be especially useful when a stable class, ID, or parent container is available. Relative methods are preferable when the anchor is discovered dynamically.
Parser choice changes the tree
BeautifulSoup can parse with Python’s built-in html.parser, lxml, or html5lib. The same malformed HTML can produce different trees under different parsers, which changes sibling and descendant relationships. Specify the parser deliberately:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")For surprising results, print a small container with prettify() and inspect the actual parent/child structure:
container = soup.select_one("dl")
print(container.prettify() if container else "container not found")The Beautiful Soup documentation currently covers Beautiful Soup 4.15.0 and explains these traversal methods at the official documentation.
Common failures and fixes
AttributeError: 'NoneType' object has no attribute ...: the anchor was not found. Check the selector, text normalization, and whether the content is present in the downloaded HTML.next_siblingreturns whitespace: this is normal. Usefind_next_sibling("tag")or loop until a tag matches.- The wrong later value is selected:
find_next()crossed into another section. Search within a parent container or stop at a boundary. - No result despite visible browser content: the HTML fetched by Python may not contain JavaScript-rendered data. Retrieve an embedded data endpoint or use a browser-rendering capture workflow.
- Different results after changing parsers: malformed markup was repaired differently. Keep the parser fixed and inspect the resulting tree.
- Text contains labels or duplicated spaces: narrow the selected node and use
get_text(" ", strip=True)orstripped_strings.
Performance, reliability, and maintainability
Prefer a specific anchor, parent scope, and tag/class filter. A bounded sibling lookup avoids scanning unrelated page content and is easier to audit when markup changes. For repeated records, locate each record container first, then perform relative searches inside that container rather than searching from the document root for every field.
Treat missing values as an expected case: return None, an empty list, or a structured error according to your application’s contract. Add tests for whitespace, nested tags, missing labels, repeated labels, and malformed HTML. If a site changes its markup, inspect the parsed tree before changing traversal code.
Or skip the browser setup
If your real goal is to obtain a clean image or PDF of a page before parsing or review, ScreenshotNeo provides a single-call website screenshot API. It accepts cookie and consent banners like a visitor, then removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each cleanup step can be disabled. Bot checks, CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and the response identifies the page verdict and billing status in X-Page-Verdict and X-Billed headers.
Use the API documentation at screenshotneo.com/docs/ for all options. A cURL request is:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpThe same request in Python:
import requests
r = requests.get(
"https://api.screenshotneo.com/v1/shot",
params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
timeout=90,
)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)And in Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const data = Buffer.from(await res.arrayBuffer());
await import('node:fs/promises').then(fs => fs.writeFile('shot.webp', data));ScreenshotNeo also offers an MCP server with take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients. Features include full-page and element capture, device and retina settings, PDF controls, custom CSS and JavaScript, waits, request blocking, cookies and headers, geolocation, caching, signed links, asynchronous webhooks, bulk capture, and a usage API. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.
Frequently asked questions
Does find_next_sibling() search inside the next sibling?
No. It searches later elements that share the anchor’s parent. To search descendants, first select the sibling and then call find() or select_one() on it.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →What is the difference between find_next_sibling() and find_next()?
The sibling method stays at one tree level. find_next() follows document order and can cross parents and nesting boundaries.
How can I preserve line breaks?
Use get_text("n", strip=True) or iterate over stripped_strings, depending on whether you need one string or separate fragments.
Frequently Asked Questions
Can I select a value between two known headings?
Yes. Find the first heading, iterate through its bounded next_elements, and stop when the second heading or another defined boundary is reached.
Why does BeautifulSoup return different siblings with different parsers?
HTML repair differs among html.parser, lxml, and html5lib. Use one explicit parser and inspect the resulting tree.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
