Start with a normal HTTP request and inspect its response. If the data you need is already there, parse it; if it appears only after JavaScript runs in a browser, use browser automation such as Playwright or Selenium. In either case, wait for evidence that the specific data is ready, then validate what you extracted.
First check whether you need a browser
A Python HTTP client such as Requests fetches HTTP responses; it does not execute a webpage’s JavaScript. A page may therefore look empty to Requests even though a browser eventually displays its content.
- Fetch the page with Requests and inspect the response body for the target text or structured data.
- If the data is present, parse that response with an HTML or data parser. This is usually simpler than starting a browser.
- If the data is absent and the page creates it client-side, use browser automation to let the page run.
This distinction matters: a browser is not required just because a site uses JavaScript. What matters is whether the response you can fetch already contains the information you need.
Choose Playwright or Selenium
Both Playwright and Selenium can automate a browser and interact with rendered pages. Neither is universally the better choice: use the API and runtime your project can support, and choose based on the way you need to wait, interact, or run the browser.
#1 Best Overall
| Consideration | Playwright | Selenium |
|---|---|---|
| Page interaction and extraction | Page-context JavaScript evaluation, locators and network monitoring are documented in the evaluation, locator and network guides. | Uses the WebDriver model; consult the Python API documentation for supported APIs and setup. |
| Waiting for readiness | Its navigation guide explains why navigation completion may precede application content readiness: Navigations. | Documents explicit and implicit wait strategies in its waiting strategies guide. |
| Browser and execution environment | Verify the current installation and browser requirements in the official Playwright documentation before deployment. | The current Python API documentation displayed version 4.50.0 and stated Python 3.10+ support when consulted for this article. It lists Chrome, Edge, Firefox, Safari, WebKitGTK, WPEWebKit and the Remote protocol; Selenium Manager handles driver and browser setup on most supported platforms with modern Selenium versions. Check the current documentation for your version and platform. |
Scrape rendered content with Playwright
Install Playwright and its browser binaries in the Python environment where the script will run:
python -m pip install playwright
python -m playwright install chromiumThe following synchronous example is illustrative. Replace the URL and the site-specific selector with the page and element that actually contain your data. The selector is not guaranteed to fit any particular site, and navigation completing does not by itself prove that the target content is ready.
from playwright.sync_api import sync_playwright
url = "https://example.com"
selector = "main article" # Replace with a selector for the target content.
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
try:
page.goto(url)
target = page.locator(selector)
target.wait_for(state="visible")
rendered_text = target.inner_text()
if not rendered_text.strip():
raise ValueError("The target element is present but contains no text")
print(rendered_text)
finally:
browser.close()For production, also handle navigation and timeout errors and validate the result against the page’s expected structure. If the page updates in place after a click, wait for the resulting element or response rather than assuming that the click caused a new navigation. Playwright’s page.evaluate() can run JavaScript in the page context and return a value to Python; pass Python values as explicit arguments because Python and page JavaScript execute in separate environments.
Wait for the state you need
A load event is a navigation milestone, not a promise that all application data has arrived. The Playwright navigation guide notes: “Modern pages perform numerous activities after the ‘load’ event was fired. They fetch data lazily, populate UI, load expensive resources, scripts and styles after the ‘load’ event was fired.”
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Prefer a condition tied to your task: a locator becomes visible, target text appears, the URL changes to the expected value, or the request that supplies the data returns. Fixed sleeps can be useful for diagnosing a timing issue, but they are a poor primary synchronization method: too short and the data is missing; too long and every run wastes time.
Extract and check the result
Use a stable locator for the relevant region rather than grabbing the whole page when possible. Then check that the output is non-empty and belongs to the intended page or query. For collections, verify the expected fields and record count and determine whether pagination or “load more” interactions are needed. Successful navigation alone is not evidence that extraction succeeded.
Use Selenium when WebDriver fits your environment
Selenium is a reasonable choice when your project already uses WebDriver or its browser and execution setup suits your deployment. Install Selenium in your Python environment and use the official Python API documentation for current driver, browser and version requirements. The documentation currently describes Selenium Manager for driver and browser setup on most supported platforms; verify that this applies to your chosen Selenium version and runtime.
Use explicit waits for a target condition instead of treating navigation as proof that the application is ready. Selenium’s waiting strategies documentation describes explicit and implicit waits. Avoid relying on a long global implicit wait as a substitute for understanding which specific element or state should appear.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
For a managed or distributed environment, Selenium’s Python API documentation lists a Remote protocol. Whether remote execution is appropriate depends on your deployment needs; the existence of the protocol does not establish that a particular hosted provider is suitable.
Inspect network traffic when the DOM route is unclear
If a page’s visible content is hard to locate or changes dynamically, inspect browser developer tools or automation network events while loading the page and performing the relevant interaction. Look for the XHR or fetch request that returns the data. Playwright documents monitoring HTTP and HTTPS traffic and waiting for responses in its network guide.
When a normal, accessible endpoint supplies the needed data and the site’s rules permit its use, calling that endpoint directly can be simpler than repeatedly rendering the full page. Confirm the request parameters, response shape and pagination rather than assuming one response contains every result. Network inspection is a way to understand how a page works; it does not itself establish permission to access or reuse the endpoint.
Common problems and fixes
- The HTML response has no target data. Requests does not execute page JavaScript. Check whether a browser-rendered page or a permitted data endpoint provides the content.
- The script runs but returns an empty result. Confirm that the selector matches the intended element and wait for the actual content, not just navigation. The page may load data lazily or update after an interaction.
- A click appears to do nothing. The application may not have attached its event handler yet, or the control may be different from the one selected. Wait for a meaningful ready condition, verify the locator, and check whether the click triggers a URL change, DOM update or network response.
- The page updates without navigating. Do not wait only for a navigation event. Wait for the updated locator, expected text or relevant response instead.
- Some records are missing. Inspect pagination, filters and “load more” behavior; validate the response and record count against what the page indicates.
- The script times out or the browser cannot start. Check the current library and browser installation requirements, confirm the required browser is installed, and increase a timeout only when the page legitimately needs more time. A longer timeout does not fix a wrong selector or an inaccessible target.
Reliability, performance and responsible access
Browser automation runs a full browser page and is more involved than parsing an HTTP response that already contains the data. Keep the simple HTTP route when it works; use browser rendering only where client-side execution or interaction is needed. Waiting on the specific target condition also avoids both premature extraction and unnecessary fixed delays.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Neither browser automation nor a direct endpoint guarantees access, completeness or permission. Check the site’s terms, access controls and applicable law before collecting data. These tools do not guarantee that bot checks can be bypassed, and no general speed or success-rate comparison between Playwright and Selenium is established here.
Or skip the browser setup
If your goal is a screenshot rather than structured data extraction, ScreenshotNeo can capture a page with one API request. Its capture steps can accept cookie or consent banners and remove more than 60 known consent platforms, newsletter popups and chat widgets; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed, and the response identifies the page verdict and billing status. An MCP server offers take_screenshot, get_page_info and capture_pdf for AI agents and other MCP clients.
For example, save a screenshot of Stripe as WebP with cURL (replace YOUR_API_KEY with your key):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo API documentation for request options. Python and Node.js alternatives:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000 screenshots. Sign up for ScreenshotNeo’s free plan.
Frequently Asked Questions
Can Requests scrape a JavaScript-rendered website?
Only if the needed data is present in the HTTP response it fetches. Requests does not run the page’s JavaScript.
Does browser automation guarantee access to a page?
No. It does not guarantee access or permission, and it is not a promise that bot checks will be bypassed.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Free tools Windows power users keep installed
One-click scans. No signup required.
