Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesUse Pyppeteer to launch Chromium, wait for the page, select the div with a CSS selector, and read its textContent. The shortest reliable pattern is page.querySelectorEval(); use an element handle when you need clearer no-match handling, and querySelectorAllEval() when several divs must be extracted.
Install Pyppeteer and Chromium on Linux
Pyppeteer is an unofficial Python port of Puppeteer for controlling headless Chrome or Chromium. Install it in the Python environment that will run your scraper:
python3 -m pip install pyppeteerOn first use, Pyppeteer can download a compatible Chromium build. You can trigger that download explicitly before running your program:
pyppeteer-installThe default Linux browser-data directory is /home/<username>/.local/share/pyppeteer; when XDG_DATA_HOME is set, Pyppeteer uses $XDG_DATA_HOME/pyppeteer. Documentation from different release eras estimates the initial Chromium download at approximately 100 MB or 150 MB, so leave additional disk space rather than treating either figure as a guaranteed current size.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
#1 Best Overall
For a server, also verify that the account running Python can write to its home or configured data directory and that the operating system has the libraries required by Chromium. A missing shared library normally appears in the browser-launch error rather than as a selector error.
Extract one div’s text
This complete script navigates to a URL, waits for network activity to settle, selects the first matching div, trims its text, prints it, and always closes Chromium:
import asyncio
from pyppeteer import launch
async def extract_div_text(url: str, selector: str) -> str:
browser = await launch(headless=True)
try:
page = await browser.newPage()
await page.goto(url, {"waitUntil": "networkidle2"})
return await page.querySelectorEval(
selector,
"node => node.textContent.trim()"
)
finally:
await browser.close()
print(asyncio.get_event_loop().run_until_complete(
extract_div_text("https://example.com", "div.article")
))querySelectorEval evaluates the supplied function against the first element that matches the CSS selector. In this example, the browser returns the div’s raw DOM textContent, and Python receives a string. The trim() call removes whitespace at the beginning and end; remove it if those characters are meaningful to your data.
Use a stable selector
Prefer an ID, stable class, or data attribute controlled by the site, such as #article-body, div.article, or [data-testid="article-body"]. Avoid selectors based on generated CSS-module names or deeply nested positional paths, because a harmless layout change can invalidate them.
Wait for a dynamically rendered div
networkidle2 waits for the navigation to become quiet, but a JavaScript application can still render the target afterward. Wait for the selector before evaluating it:
await page.goto(url, {"waitUntil": "networkidle2"})
await page.waitForSelector("div.article")
text = await page.querySelectorEval(
"div.article",
"node => node.textContent.trim()"
)Choose a selector that represents the content being ready, not merely a wrapper that appears before its text is populated. If the page intentionally loads content after an interaction, perform that interaction first and then wait.
Handle a missing element without losing the browser
querySelectorEval raises when no element matches. Treat that as an expected data condition when URLs are heterogeneous, and report the URL and selector:
from pyppeteer.errors import ElementHandleError
async def extract_optional(page, selector: str):
try:
return await page.querySelectorEval(
selector,
"node => node.textContent.trim()"
)
except ElementHandleError:
return NoneThe exact exception presentation can vary by Pyppeteer version, so a broader exception boundary may be appropriate at an application boundary, with logging that preserves the original traceback. Do not silently convert browser crashes, navigation failures, and selector misses into the same empty string; they require different recovery decisions.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Element-handle alternative
For explicit control, first obtain an element handle and then evaluate against it:
element = await page.querySelector("div.article")
if element is None:
raise ValueError("No element matched div.article")
text = await page.evaluate(
"(element) => element.textContent",
element
)
text = text.strip()This separates “find” from “read,” making it easier to add diagnostics, inspect attributes, or choose a fallback selector. It also makes the no-match branch obvious.
Extract text from every matching div
Use querySelectorAllEval when the selector can match multiple elements. The callback receives an array of nodes and maps each one to trimmed text:
texts = await page.querySelectorAllEval(
"div.article",
"nodes => nodes.map(node => node.textContent.trim())"
)
for text in texts:
print(text)The result preserves document order. An empty list means the selector matched nothing; it is not the same as a list containing an empty string, which indicates a matching div with no text content.
Choose between textContent and innerText
Both properties can be evaluated through the same selector APIs:
raw_dom_text = await page.querySelectorEval(
"div.article", "node => node.textContent"
)
rendered_text = await page.querySelectorEval(
"div.article", "node => node.innerText"
)textContent reads the DOM text associated with the element, while innerText is intended for rendered, user-visible text and can reflect layout and visibility. Their whitespace and hidden-content behavior can differ. Use textContent when you need the page’s underlying text nodes; choose innerText when your requirement is closer to what a visitor sees. Normalize whitespace in Python only after deciding which semantic you need.
Pyppeteer selector names on Python
JavaScript Puppeteer examples often use $, $$, and $x. Python cannot use those method names directly, so Pyppeteer exposes querySelector, querySelectorAll, and Jx/J/JJ shorthands, plus the explicit querySelectorEval and querySelectorAllEval methods shown above. The explicit names are usually clearest in shared code and documentation.
Navigation, timing, and cleanup patterns
Use the right navigation wait
waitUntil="networkidle2" is useful for pages that finish loading after a small number of outstanding requests. Analytics, advertisements, WebSockets, and long polls can prevent a true idle state or make it occur before the application has rendered its final content. Pair navigation with waitForSelector for the data you actually need.
Best Value
Always close the browser
Put extraction inside try and browser shutdown in finally. This prevents orphaned Chromium processes when navigation, evaluation, or parsing fails. For batch jobs, consider one browser with multiple pages rather than launching a new browser for every URL, while still closing every page and browser at the end of the job.
Evaluate expressions correctly
If an expression such as document.body.textContent is interpreted as a JavaScript function rather than an expression, call page.evaluate(..., force_expr=True). For element extraction, an arrow function such as node => node.textContent avoids that ambiguity.
Common failures and fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Chromium will not launch | First-run browser download, permissions, or missing Linux libraries | Run pyppeteer-install, check the Pyppeteer data directory, and inspect the complete Chromium error for missing dependencies. |
querySelectorEval raises |
No element matches at evaluation time | Verify the selector, wait for it with waitForSelector, or use a guarded element-handle branch. |
| Result is empty | The div exists but has no text nodes, or content is inserted later | Wait for the content-bearing selector or application state; inspect whether the desired value is in an attribute rather than text. |
| Only some pages work | Different templates use different selectors | Define per-template selectors or ordered fallbacks and record which selector matched. |
| Navigation times out | Slow resources, blocked requests, or a page that never becomes idle | Use a suitable timeout and wait condition, then separately wait for the required selector instead of relying only on network idle. |
| Text differs from the visible page | textContent includes DOM text that is not presented as rendered text |
Test innerText for a rendered-text requirement and document the choice. |
Production considerations
- Pin versions: Pyppeteer is described by its project as unmaintained and recommends evaluating Playwright Python for new work. Pin your Python and browser versions if you must maintain a Pyppeteer integration.
- Control concurrency: Chromium is resource-intensive. Limit simultaneous pages, reuse a browser where safe, and measure memory before increasing parallelism.
- Log context: Record URL, selector, navigation outcome, wait condition, and whether zero, one, or many elements matched.
- Respect access rules: Follow the target site’s terms, robots guidance, authentication requirements, and rate limits. Do not treat headless browsing as permission to bypass access controls.
- Cache deliberately: If content changes slowly, cache successful results and retry transient navigation errors with backoff rather than repeatedly launching browsers.
Or skip the browser setup
If your goal is a clean screenshot rather than DOM text, ScreenshotNeo provides a website screenshot API and MCP server. A single request captures a URL as PNG, JPEG, WebP, or PDF:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);See the ScreenshotNeo documentation for request options. Before capture, it can accept consent banners and remove more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be disabled. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing result. Its MCP server includes take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients. The Free plan includes 1,000 screenshots per month without a card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.
Frequently Asked Questions
Can Pyppeteer extract text from a div inside an iframe?
Not from the parent page context directly. Obtain the iframe’s frame, wait for the selector in that frame, and run the same evaluation against the frame.
Why does my selector work in DevTools but not in Pyppeteer?
The element may be inside an iframe, shadow DOM, or a later rendering state. Confirm the browsing context and wait for the element after navigation.
Should a new project still use Pyppeteer?
Pyppeteer remains useful for compatibility with existing code, but its repository describes it as unmaintained; evaluate a maintained alternative such as Playwright Python for new production systems.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Free tools Windows power users keep installed
One-click scans. No signup required.
