Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
If a site blocks headless Playwright, first check whether you are allowed to automate access. Prefer its API, export, or documented integration; if browser access is permitted, test Playwright’s real-Chrome headless mode and make your crawler stable, low-rate, and easy to identify. If the site presents a challenge, rate limit, login wall, or block, do not try to evade it: request permission or stop. A different browser mode can improve compatibility, but it cannot grant access or guarantee a successful scrape.
Why a site can detect headless Playwright
There is no single “headless” flag that explains every block. Anti-bot systems can combine browser signals with request, session, network, and behavioral context. Cloudflare describes using multiple detection engines because different bot types require different strategies; its documentation covers heuristic checks, JavaScript detections, machine-learning scores, and scraping detections.
Browser and JavaScript signals
A site can evaluate browser APIs, rendering behavior, and inconsistencies in how the browser presents itself. Cloudflare says its JavaScript Detections engine identifies headless browsers and other malicious fingerprints. That does not mean every headless browser will be blocked, or that a single browser signal always decides the outcome; systems may assess several signals together.
Request, session, and network context
Headers, cookies, session history, browser signals, network characteristics, and request patterns can all contribute. Cloudflare describes a machine-learning bot score from 1 to 99 and says its scraping detections analyze ASN and JA4 patterns, with suspicious traffic scores recalculated dynamically. Turnstile also says it evaluates client and site signals, including network- and browser-related signals.
#1 Best Overall
- CRISP CLARITY: This 23.8″ Philips V line monitor delivers crisp Full HD 1920x1080 visuals. Enjoy movies, shows and videos with remarkable detail
- INCREDIBLE CONTRAST: The VA panel produces brighter whites and deeper blacks. You get true-to-life images and more gradients with 16.7 million colors
- THE PERFECT VIEW: The 178/178 degree extra wide viewing angle prevents the shifting of colors when viewed from an offset angle, so you always get consistent colors
- WORK SEAMLESSLY: This sleek monitor is virtually bezel-free on three sides, so the screen looks even bigger for the viewer. This minimalistic design also allows for seamless multi-monitor setups that enhance your workflow and boost productivity
- A BETTER READING EXPERIENCE: For busy office workers, EasyRead mode provides a more paper-like experience for when viewing lengthy documents
As a result, changing only the user-agent or adding delays may not address the reason for a block. Nor does the fact that a page loads in your everyday browser establish that an automated client is permitted to access it.
Challenges and denials are access controls
A JavaScript challenge, Turnstile check, 403 or 429 response, authentication prompt, or explicit block can mean the site is restricting automated access. Treat these as meaningful responses, not merely rendering errors to work around. Stop or ask the site owner for an approved route rather than escalating your automation.
Check authorization before changing your browser
Use the least intrusive permitted option, in this order:
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteRank #2
- CRISP CLARITY: This 22 inch class (21.5″ viewable) Philips V line monitor delivers crisp Full HD 1920x1080 visuals. Enjoy movies, shows and videos with remarkable detail
- 100HZ FAST REFRESH RATE: 100Hz brings your favorite movies and video games to life. Stream, binge, and play effortlessly
- SMOOTH ACTION WITH ADAPTIVE-SYNC: Adaptive-Sync technology ensures fluid action sequences and rapid response time. Every frame will be rendered smoothly with crystal clarity and without stutter
- INCREDIBLE CONTRAST: The VA panel produces brighter whites and deeper blacks. You get true-to-life images and more gradients with 16.7 million colors
- THE PERFECT VIEW: The 178/178 degree extra wide viewing angle prevents the shifting of colors when viewed from an offset angle, so you always get consistent colors
- Look for an official API, feed, export, or documented integration. These are more stable and predictable than scraping page markup, and often explain permitted use and rate limits.
- Review the site’s terms and robots.txt. Read the rules that apply to your intended use, not only whether a URL can be fetched. Cloudflare explains that robots.txt compliance is voluntary: the file communicates a site’s preferences but does not technically prevent access. That technical fact is not permission to disregard the file or the site’s terms.
- Ask for permission or allowlisting if automation is blocked. Tell the site owner what you need, why, and how often you will access it. If requested, provide your user-agent, source IP ranges, schedule, and endpoints. Use an API key or verified-bot route if the owner offers one.
- Set conservative operating limits. Keep concurrency low, cache responses, and avoid fetching pages you do not need. Follow documented quotas and any instructions from the site owner.
- Stop at a challenge or denial. Do not automate CAPTCHA solving, disguise traffic, rotate identities, or repeatedly retry access after the site has refused it.
Terms and legal rules differ by site, use, and jurisdiction. Cloudflare’s sample terms include a restriction on automated scraping and AI-related use unless specified conditions are met, but the sample is informational and explicitly not legal advice. For commercial, sensitive, or otherwise high-risk collection, review the target’s actual terms and obtain jurisdiction-specific advice.
Try Playwright’s real-Chrome headless mode for authorized pages
If you have permission to automate a page, test Playwright’s documented chromium channel. Playwright says this channel uses Chrome’s new headless mode and quotes Chrome’s description of it as “the real Chrome browser.” That can reduce compatibility differences between headless automation and ordinary Chrome behavior. It is a compatibility option, not a bypass: it does not override site policy or guarantee that a challenge will be passed.
Minimal Node.js example
Install Playwright and its Chromium browser in a project, then save the following as scrape.js. Replace the example URL and selector with a page and content you are authorized to access. The example visits one page, checks the HTTP response, extracts text from a known selector, and closes the browser.
Rank #3
- Clear visuals. Fluid motion: A 144Hz refresh rate and 1ms MPRT deliver smooth, tear‑free motion across work, gaming, and streaming for clearer, more fluid viewing.
- Eye comfort: TÜV Rheinland 3‑star* certification reduces harmful blue light while preserving stunning color quality without compromise. *TÜV Rheinland 3-star eye comfort certification.
- Wide viewing angle: Get consistent views across a wide 178° /178° viewing angle.
- In-Plane Switching (IPS): See excellent color accuracy and consistency across wide viewing angles with In-plane Switching (IPS) technology.
- Ultra-thin bezels: Maximize your viewing experience with thin bezels.
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch({
channel: 'chromium',
headless: true
});
try {
const page = await browser.newPage();
page.setDefaultNavigationTimeout(30000);
const response = await page.goto('https://example.com/', {
waitUntil: 'domcontentloaded'
});
if (!response) {
throw new Error('Navigation returned no response; check the URL and network.');
}
const status = response.status();
if (status === 403 || status === 429) {
throw new Error(`Access refused or rate limited (HTTP ${status}); stop and seek permission.`);
}
if (!response.ok()) {
throw new Error(`Page returned HTTP ${status}.`);
}
const selector = 'main';
await page.locator(selector).waitFor({ state: 'visible', timeout: 10000 });
const text = await page.locator(selector).innerText();
console.log(text);
} catch (error) {
console.error(error.message);
process.exitCode = 1;
} finally {
await browser.close();
}
})();For an authorized target, a practical installation and run sequence is npm install playwright, npx playwright install chromium, then node scrape.js. The selector is deliberately site-specific: inspect the page you are permitted to collect from and choose a stable content container, rather than assuming every site uses main.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Choose waits based on the page
domcontentloaded: a reasonable starting point when the needed markup is available early; it avoids waiting for every image or third-party resource.- A selector wait: use
locator(selector).waitFor()when the content appears after client-side rendering. Set a finite timeout and handle the case where it never appears. networkidle: use cautiously. Analytics, chat, and other long-lived requests can keep a page active, making this wait slow or unreliable. Wait for the specific content you need when possible.
Do not treat a timeout as a reason to increase concurrency or evade a challenge. Determine whether the page is simply slow, the selector changed, access requires authentication, or the site denied the request.
Handle blocks and failures without escalating
Log enough to diagnose authorized jobs: target URL, timestamp, response status, elapsed time, timeout or selector error, and whether the response looked like expected content. Do not log passwords, access tokens, or private page data unnecessarily.
Rank #4
- CURVED FOR ENHANCED ENGAGEMENT: An immersive viewing experience with a curved monitor that wraps more closely around your field of vision; It creates a wider view, enhancing depth perception and minimizing peripheral distraction
- SMOOTH PERFORMANCE FOR SEAMLESS CONTENT: Stay in the action when playing games, watching videos, or working on creative projects; The 100Hz refresh rate reduces lag and motion blur so you don't miss a thing in fast-paced moments¹
- MORE GAMING POWER: Gain the edge with optimizable game settings; Color and image contrast can be adjusted to see scenes more vividly and spot enemies hiding in the dark; Game Mode adjusts any game to fill the screen so you can view every detail²
- KEEP IT EASY ON THE EYES: Care for your eyes and stay comfortable, even during long sessions; Advanced eye comfort technology certified by TÜV reduces eye strain by minimizing blue light and reducing irritating screen flicker²
- INCREASED VERSATILITY: Connect to more; Plug devices straight into your monitor for increased flexibility, making your computing environment even more convenient
403, 429, or a challenge page
A 403 commonly indicates that the server refused the request; a 429 indicates rate limiting. A challenge page may return a successful HTTP status while showing no target content. On these outcomes, do not keep retrying or switch identities to get through. Respect a supplied Retry-After value for an authorized rate-limited integration, reduce request frequency, and contact the site owner if the problem persists. If the owner has not authorized continued access, stop.
Timeout or missing selector
First confirm the URL is correct and reachable under the approved access method. Then check whether the target content is rendered later or whether the selector changed. Use an explicit selector wait with a reasonable timeout rather than waiting indefinitely for all network activity. If a login or consent step is required, use only an owner-approved session or process.
Blank or incomplete content
Check whether the intended text exists in the rendered page and whether the selected container is correct. Client-side rendering may require waiting for a specific element. A blank page can also indicate a failed load, challenge, or unavailable content; distinguish those cases in logs and do not interpret an empty capture as a successful scrape.
Best Value
- 【INTEGRATED SPEAKERS】Whether you're at work or in the midst of an intense gaming session, our built-in speakers provide rich and seamless audio, all while keeping your desk clutter-free.
- 【EASY ON THE EYES】 Protect your eyes and enhance your comfort with Blue-Light Shift technology. This feature reduces harmful blue light emissions from your screen, helping to alleviate eye strain during long hours of use and promoting healthier viewing habits.
- 【WIDEN YOUR PERSPECTIVE】Our sleek minimal bezel design ensures undivided attention. The nearly bezel-free display seamlessly connects in a dual monitor arrangement, delivering an unobstructed view that lets you focus on more at once, completely distraction-free.
Repeated failures
Back off rather than immediately retrying; cache successful results; and reduce concurrency. If the site continues to refuse requests, stop and ask about an allowlist, API, or approved schedule. Do not treat proxy rotation, randomized delays, user-agent changes, or stealth patches as a legitimate fix. Cloudflare’s bot-protection guide describes such evasion behaviors in the context of attackers, and they can make traffic appear more suspicious.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Choose an approach based on what you need
For structured fields or recurring collection, an official API or export is usually the best fit when available and authorized. Permissioned Playwright makes sense when the approved source exposes data only through a rendered page. A hosted browser service can be useful for managed rendering, but it does not remove a target site’s controls: Cloudflare Browser Run documents Playwright support and says target-site bot controls still apply to its crawler.
Compare options by authorization, data completeness, JavaScript needs, stability when the site changes, allowed request volume, observability, cost, and whether the provider can arrange allowlisting. A browser renderer returns a rendered page or image; it does not automatically provide the structured records a scraper needs. Whatever the method, an owner’s permission and the site’s terms govern access.
Or skip the browser setup
If the goal is a clean visual record rather than structured data, ScreenshotNeo can return a screenshot or PDF with one GET request. It is a screenshot API, not a way to extract structured page data or bypass a target site’s bot controls. Its capture options include PNG, JPEG, or WebP output, PDF, full-page capture, and waiting for a selector, delay, or network idle. See the ScreenshotNeo API documentation for parameters and response details.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com -o shot.webp- Before capture, it accepts the cookie or consent banner like a visitor and removes 60+ known consent platforms, newsletter popups, and chat widgets; each step can be turned off.
- Bot checks, CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing; response headers identify the page verdict and billing status.
- An MCP server offers
take_screenshot,get_page_info, andcapture_pdftools for Claude, Cursor, and other MCP clients. - The Free plan includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots.
Sign up for ScreenshotNeo’s free plan to get 1,000 screenshots a month with no card.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
