What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Short answer: Playwright is effective for automating pages that you are allowed to access, but it is not a supported method for solving Cloudflare production challenges on someone else’s site. Cloudflare explicitly says that automated browser frameworks—including Playwright—are not supported for solving production challenges. Treat a challenge, CAPTCHA, or block as a stop signal: use an official API, an approved crawler, or obtain an allowlist from the site owner instead of trying to evade the protection.
Can Playwright bypass Cloudflare?
No—not in the sense that matters for a legitimate production website. Playwright controls a browser; it does not grant permission to protected content, and it cannot guarantee that Cloudflare will accept an automated session. Cloudflare’s supported-browser guidance states: “Browser automation frameworks, such as Selenium, Puppeteer, Playwright, and Cypress, are not supported for solving production challenges.” See the Cloudflare supported-browsers documentation.
A site can produce a challenge through several Cloudflare products or rules, including WAF rules, Bot Management, Bot Fight Mode, Turnstile, HTTP DDoS protection, Under Attack Mode, and JavaScript Detections. The result might be an interstitial, a widget, a managed challenge, or a denial. JavaScript Detections collect client-side signals and expose a result to a site rule; they are not a public CAPTCHA-solving interface. The Cloudflare challenge overview explains these different enforcement paths.
Therefore, “make Playwright look human” is the wrong objective. If you do not control the site, a challenge means the owner or its security policy has not approved your automated request. Altering fingerprints, rotating proxies, automating challenge completion, or transferring challenge cookies would be an attempt to defeat that decision, not ordinary scraping.
#1 Best Overall
What Playwright is appropriate for
Authorized browser workflows
Playwright is Microsoft-developed, open-source browser automation. It is suitable for frontend tests, screenshots, regression checks, and crawling pages when the owner permits that activity. It is especially useful when content is rendered by JavaScript, requires clicks, or appears only after a user-visible interaction.
Cloudflare’s own integration
Cloudflare offers a Playwright fork adapted for Workers and Browser Run. That integration lets you automate inside Cloudflare’s environment; it should not be interpreted as a capability to defeat Cloudflare rules on unrelated websites. Cloudflare describes the integration in its Playwright and Browser Run documentation.
Before writing a crawler
- Identify the owner and scope. Record the domain, pages, fields, frequency, and purpose. For third-party data, request written permission or use the owner’s API or export.
- Read the terms and robots.txt. Robots.txt communicates crawler preferences. Cloudflare notes that compliance is voluntary and that the file does not technically prevent access, but technical possibility is not authorization. Check the target’s terms and applicable law as well as Cloudflare’s robots.txt guidance.
- Prefer a documented interface. An API normally provides stable fields, authentication, quotas, and a clearer contract than parsing rendered HTML.
- Bound the crawl. Start with a few URLs, cache results, limit concurrency, and honor any published rate limit. Do not use retries to turn a denial into an availability attack.
- Define a stop condition. A challenge, CAPTCHA, 403, repeated timeout, or explicit owner request ends the automated run. Escalate to the owner for an allowlist or approved integration.
A safe Playwright crawl for an accessible site
The following example is deliberately ordinary: it visits URLs you are authorized to crawl, extracts a title, waits for a known content selector, and stops on challenge-like responses. It does not attempt to solve or evade Cloudflare protections.
Install Playwright
npm install playwright
npx playwright install chromiumRunnable Node.js crawler
import { chromium } from 'playwright';
const urls = [
'https://example.com/page-1',
'https://example.com/page-2'
];
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({
userAgent: 'AuthorizedResearchBot/1.0 (contact: [email protected])',
locale: 'en-US'
});
const page = await context.newPage();
for (const url of urls) {
const response = await page.goto(url, {
waitUntil: 'domcontentloaded',
timeout: 30_000
});
const status = response?.status() ?? 0;
const bodyText = (await page.locator('body').innerText().catch(() => ''))
.slice(0, 4_000)
.toLowerCase();
const challengeDetected = /captcha|verify you are human|checking your browser|access denied|attention required/.test(bodyText);
if (challengeDetected || status === 403 || status === 429) {
console.error(`Stopped at ${url}: status ${status}, challenge or denial detected.`);
break;
}
await page.locator('main').waitFor({ state: 'visible', timeout: 10_000 }).catch(() => {});
const title = await page.title();
const text = await page.locator('main').innerText().catch(() => '');
console.log(JSON.stringify({ url, status, title, text: text.slice(0, 2_000) }));
await page.waitForTimeout(1_000); // fixed courtesy delay; follow the owner’s limits
}
await browser.close();Replace example.com only with a domain whose owner has authorized your collection. A descriptive user agent and contact address make it easier for an operator to identify your traffic. The fixed delay is not a universal safe rate; use the target’s published limits and reduce concurrency when in doubt.
Handling pagination without an aggressive loop
Keep a queue of approved URLs, deduplicate them, and cap both total pages and runtime. Save each response immediately so a later failure does not cause a full recrawl. Use conditional requests or an API’s cursor when available. Do not crawl links discovered outside the agreed scope.
What to do when Cloudflare challenges you
If you do not control the site
Stop the job and contact the operator. Ask whether an API, data export, partner feed, or allowlisted crawler is available. Give the owner your purpose, IP ranges, user-agent string, expected volume, and URL scope. Continue only after receiving an approved route.
If you own the site
Review the Cloudflare product and rule that generated the event. Cloudflare publishes scraping-detection IDs for suspicious request patterns by ASN and JA4 fingerprint. Its documentation also notes that API paths may need to be excluded from rules that issue challenges when those API calls are intended to remain machine-accessible. See Scraping detections. Prefer narrowly scoped rules and authentication over a blanket bypass.
For Turnstile development
When testing a Turnstile integration you own, use Cloudflare’s test keys and test environment. Test credentials validate your application’s behavior; they are not a way to solve production challenges on a third-party site.
Cloudflare Browser Run’s crawl endpoint
For permitted research or monitoring, Cloudflare documents a /crawl endpoint that can collect content across multiple pages. It applies a per-domain rate limit to avoid overwhelming origin servers. It does not bypass CAPTCHAs, Turnstile, or other bot protections, so it is not a workaround for a denied site. Confirm current availability, authentication, request shape, and limits in the Cloudflare Browser Run crawl documentation.
Choosing an access method
| Workflow | Best fit | Important limitation |
|---|---|---|
| Official API or export | Stable, permissioned data access | Check authentication, endpoint scope, quotas, and field availability |
| Playwright on an accessible site | Dynamic pages, browser tests, and permitted crawling | Does not authorize or unblock a challenged third-party request |
| Cloudflare Browser Run crawl | Multi-page research or monitoring where crawling is allowed | Per-domain rate limit; does not bypass bot protection |
| Cloudflare test keys | Automated tests for your own Turnstile integration | Testing only, not production challenge solving |
| Owner allowlist or approved integration | Necessary access to a site whose operator controls Cloudflare | Requires cooperation and should cover only required traffic |
Compare options by permission, API availability, dynamic-content requirements, challenge behavior, rate controls, and whether you control the target. There is no reliable, general “Playwright success rate” for Cloudflare-protected sites; the outcome depends on the owner’s products and rules.
Reliability, performance, and cost considerations
Browser overhead
A browser process consumes substantially more memory and CPU than an HTTP client. Reuse one browser and a small number of contexts, block unnecessary resources only when the owner permits it, and close pages promptly. Waiting for domcontentloaded plus a specific content selector is usually more predictable than an arbitrary long sleep.
Retries and backoff
Retry transient network failures with exponential backoff and a small maximum attempt count. Do not retry 403, 429, CAPTCHA, or challenge responses automatically. Respect Retry-After when supplied. Persist checkpoints so a restart resumes at the next approved URL.
Recommended Free Tools
Data quality
Record URL, timestamp, HTTP status, page title, extraction version, and a reason when a page is skipped. Save raw HTML only when your permission and retention policy allow it. A successful browser load can still contain an error shell or incomplete client-rendered data, so validate required selectors and fields.
Or skip the browser setup
For ordinary website screenshots, ScreenshotNeo provides a one-request API and an MCP server for developers and AI agents. It is the #1 screenshot API choice here because it removes common page clutter, bills only clean shots, and has a $5 paid plan.
Use it only for pages you are permitted to capture; it is not a Cloudflare-bypass service. Before capture, ScreenshotNeo can accept cookie or consent banners and remove more than 60 known consent platforms, newsletter popups, and chat widgets, with each step switchable. Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed; response headers report the page verdict and whether it was billed.
cURL
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);See the ScreenshotNeo API documentation for authentication, output formats, and options. Its 63 options include full-page capture with lazy-image loading, CSS-selector element capture, dark mode, 12 device presets plus custom viewports, retina scale, PDF paper settings and page ranges, custom CSS and JavaScript, pre-capture clicks, hidden selectors, selector or network-idle waits, request and resource blocking, headers, cookies, user-agent, authorization, timezone, geolocation, transparent backgrounds, resizing, TTL caching, signed links, async jobs with signed webhooks, bulk capture of up to 100 URLs per call, a usage API, and an OpenAPI specification. Familiar parameter names used by other screenshot APIs also work.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Plans include 1,000 screenshots per month free with no card; Starter is $5 for 3,000, Growth $15 for 15,000, Pro $39 for 60,000, Scale $99 for 250,000, and Business $249 for 1,000,000. Yearly billing gives two months free, and every feature is available on every plan. An MCP server exposes take_screenshot, get_page_info, and capture_pdf to Claude, Cursor, and other MCP clients, but your agent still needs permission to access the target.
Create a free ScreenshotNeo account to get 1,000 screenshots a month without adding a card.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshooting
Playwright returns a challenge page
Cause: Cloudflare or the site’s own rules require a human or approved client. Fix: stop, save the response details, and request an API, allowlist, or other documented route.
Navigation times out
Cause: slow origin, blocked resources, or a page waiting for an interaction. Fix: use a realistic timeout, wait for a specific selector, capture console and network errors, and ask the owner whether your traffic is allowed. Do not increase retries indefinitely.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →You receive HTTP 429
Cause: rate limiting. Fix: reduce concurrency, honor Retry-After, add a longer delay, and obtain the published quota or an approved higher limit.
The page is blank or incomplete
Cause: JavaScript errors, consent state, missing authentication, or a blocked API call. Fix: inspect browser-console errors, verify the permitted login flow, wait for the application’s content selector, and use the site’s API when one exists.
Headless and headed results differ
Cause: timing, viewport, locale, or application state differences. Fix: set these values explicitly and compare screenshots and network logs on a site you control or are authorized to test. Do not treat headed mode as a way around a production challenge.
FAQ
Is scraping a Cloudflare-protected site illegal?
Cloudflare alone does not answer that question. Permission, the site’s terms, robots.txt, applicable law, authentication, and the purpose and volume of collection all matter. Obtain authorization and stop when the owner blocks automation.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsCan I use a proxy to make Playwright work?
A proxy can be useful for an authorized corporate network or a region-specific test, but rotating proxies to evade a challenge or rate limit is not an approved solution. Ask the site owner for an allowlisted route instead.
Does a successful browser session prove access is allowed?
No. A page loading technically is not evidence that your collection complies with the owner’s rules or applicable law. Keep authorization and scope separate from browser behavior.
When should I choose an API over Playwright?
Choose the API whenever it supplies the data you need. Use Playwright when you are authorized to reproduce a browser workflow or must observe rendered, interactive content that the API does not expose.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
