Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsUse a real browser context, identify what is blocking the page, and make an explicit choice. A native JavaScript dialog needs dialog.accept() or dialog.dismiss(). A DOM banner should be rejected or closed with a site-specific locator, including a frame locator when the consent manager runs in an iframe. If the site stores consent in cookies or local storage, create an isolated context and restore that state on later runs. Do not assume that disabling cookies, deleting an overlay, or clicking an arbitrary button represents valid consent.
The reliable workflow is: classify the obstruction, choose “Reject” (or another documented policy), persist the resulting state, and verify that the page remains usable. The examples below show Playwright first, then Selenium, followed by an API option when you only need a clean image or PDF.
Classify the obstruction before changing cookies
“Cookie modal” can mean three different browser behaviors. Inspect the page and browser events before writing a selector or changing privacy settings.
Native JavaScript dialogs
alert, confirm, prompt, and beforeunload are browser dialogs, not HTML. Playwright auto-dismisses dialogs by default. If you register a listener, always settle it; an unhandled dialog can pause the action that triggered it.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutepage.on('dialog', async dialog => {
console.log(`${dialog.type()}: ${dialog.message()}`);
await dialog.dismiss();
});Accept only when your policy requires the dialog’s affirmative action. For a confirmation that must be accepted, replace dismiss() with accept(). Never leave a listener that logs but does not call either method.
#1 Best Overall
DOM banners and modal overlays
Most consent notices are ordinary elements. Locate the visible dialog by an accessible role and button name, stable data attribute, or a known iframe. Prefer the site’s “Reject all”, “Necessary only”, or equivalent control. A close icon that merely hides the panel may leave consent unset.
For a predictable banner, wait for the control and click it explicitly. For an intermittent overlay, current Playwright releases provide page.addLocatorHandler(), which runs a self-contained handler before an action or assertion encounters the matching locator. The handler can change focus and mouse state, so subsequent actions should not depend on the previous focus.
Persisted consent state
Consent-management platforms commonly check cookies or local storage before displaying a notice. You can create a fresh context, make the documented choice once, save storageState, and restore it for repeatable runs. Cookie domain, path, expiry, and the meaning of each value are site-specific; never copy a cookie name or value between domains without verifying it.
Playwright: reject the banner and reuse the decision
Install Playwright and its browser, then pass a target URL on the command line:
npm install playwright
npx playwright install chromiumThis complete Node.js example handles native dialogs, an accessible DOM dialog, and a saved state. The regular expression is intentionally narrow enough to avoid clicking unrelated buttons; adapt the words to the target site.
import { chromium } from 'playwright';
const targetUrl = process.argv[2] ?? 'https://example.com';
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();
// Put site-specific pre-load logic here when the CMP requires it.
await context.addInitScript({ content: '/* site-specific hook */' });
const page = await context.newPage();
page.on('dialog', async dialog => {
console.log(`Dismissing ${dialog.type()} dialog: ${dialog.message()}`);
await dialog.dismiss();
});
await page.addLocatorHandler(page.getByRole('dialog'), async dialog => {
const reject = dialog.getByRole('button', {
name: /reject|necessary only|deny|decline/i
});
if (await reject.count()) {
await reject.first().click();
return;
}
const close = dialog.getByRole('button', { name: /close/i });
if (await close.count()) await close.first().click();
});
await page.goto(targetUrl, { waitUntil: 'domcontentloaded' });
// A site-specific assertion is better than assuming the modal is gone.
const consentDialog = page.getByRole('dialog');
if (await consentDialog.count()) {
await consentDialog.first().waitFor({ state: 'hidden', timeout: 5000 }).catch(() => {});
}
await page.screenshot({ path: 'page.png', fullPage: true });
await context.storageState({ path: 'consent-state.json' });
await browser.close();Run the first visit once, inspect the saved state, and then restore it on subsequent jobs:
const context = await browser.newContext({ storageState: 'consent-state.json' });Keep the state file private: it can contain authentication and preference data as well as consent. Regenerate it when the site changes its consent manager or when the stored expiry has passed.
When the banner is inside an iframe
A top-level getByRole('dialog') cannot see elements in another browsing context. Identify the frame by a stable name or URL fragment and use frameLocator:
const cmp = page.frameLocator('iframe[title*="consent" i]');
await cmp.getByRole('button', { name: /reject|necessary only/i }).click();If the iframe is injected late, wait for the iframe element first. Avoid broad selectors such as “the first button on the page”; they are fragile and can trigger an unrelated action.
Using an initialization hook
addInitScript runs before the page’s scripts, which is useful for a site-specific hook that reads or sets a preference before the consent manager initializes. The hook must match that site’s implementation. A generic script that removes every element containing the word “cookie” can hide legitimate content and does not prove that tracking was disabled.
Selenium alternative
Selenium WebDriver can drive Chrome in the same way, but the browser and ChromeDriver major versions must match. Selenium does not define a universal cookie-banner selector, so the locator remains site-specific.
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
options = webdriver.ChromeOptions()
options.add_argument('--headless=new')
driver = webdriver.Chrome(options=options)
try:
driver.get('https://example.com')
wait = WebDriverWait(driver, 10)
try:
reject = wait.until(EC.element_to_be_clickable((
By.XPATH,
"//button[contains(translate(normalize-space(.), 'REJECTDENY', 'rejectdeny'), 'reject') or contains(translate(normalize-space(.), 'NECESSARYONLY', 'necessaryonly'), 'necessary')]"
)))
reject.click()
except TimeoutException:
pass # No matching banner appeared; record this outcome.
# Continue only after a page-specific readiness check.
print(driver.title)
finally:
driver.quit()For a repeatable Selenium run, make the choice once, call driver.get_cookies(), and restore only the cookies whose domain, path, and expiry you have verified. Add cookies after opening the matching origin and before loading the protected page; otherwise WebDriver will reject them or the site will ignore them.
Why browser-wide cookie blocking is not a universal fix
Firefox and Safari apply stronger third-party-cookie protections by default than Chrome. Chrome generally does not block third-party cookies outside Incognito or an explicit setting. Blocking cookies can also break embedded media, payment widgets, login flows, and social components. Treat the browser setting as a diagnostic or privacy control, not as a modal-removal technique.
A consent manager may display a notice because it has no valid preference, record the response, and pass the status to analytics and advertising systems. Deleting the banner from the DOM neither records a rejection nor guarantees that tags stopped running. Make the policy decision visible in your scraper configuration and logs.
Choose the method that matches your target
| Method | Reliability across sites | Records an explicit choice | Repeatability | Main risk |
|---|---|---|---|---|
| Click “Reject” in the page DOM | Good when selectors are maintained | Usually, if the site control is genuine | Good after saving state | Selector changes or delayed rendering |
| Click inside a consent iframe | Good after frame discovery | Usually | Good | Frame URL, title, or nesting changes |
| Restore verified cookies/local storage | Very good for one stable site | Only if state came from a real documented choice | Very good until expiry or CMP changes | Wrong domain, path, or stale semantics |
| Remove the overlay with JavaScript | Low | No | Low | Tracking or layout can continue invisibly |
| Block cookies globally | Inconsistent | No | Inconsistent | Broken functionality and third-party components |
For intermittent overlays, a locator handler is convenient. For a high-volume crawl of one domain, a verified storage state is faster and less brittle. For multiple unrelated domains, keep separate contexts and state files so one site’s consent data cannot leak into another.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Make scraping repeatable and defensible
Record the decision
Log the URL, timestamp, action taken (“Reject all”, “Necessary only”, or no banner), browser version, and whether the state was restored or created. Keep the consent-state file alongside a versioned scraper configuration, not in source control if it contains other cookies.
Wait for the right event
domcontentloaded confirms that the initial document is parsed, not that a delayed CMP has finished. Use a short, site-specific wait for the dialog or its absence, then wait for the content you actually need. A blanket networkidle wait can be slow on pages with analytics or streaming requests.
Handle retries without duplicating consent
Reuse the same context for related pages on one domain. If navigation fails, retry the page load rather than clicking the consent control repeatedly. On a new context, expect the banner again unless a valid state is supplied.
Respect access and privacy rules
Check the site’s terms, robots directives, and applicable privacy law. Do not treat a hidden banner as proof of consent, and do not collect personal data merely because the browser can display it. “Reject” should be your documented default where it is available and consistent with your legal basis.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Common failures and fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Locator times out | Banner is delayed, renamed, or inside a frame | Wait for the container, inspect frames, and use a stable role, text, or data attribute. |
| Click is intercepted | Another overlay, animation, or focus trap is active | Use a self-contained locator handler, wait for visibility, and avoid force-clicking until the cause is understood. |
| Banner returns every run | Consent was not stored, storage is partitioned, or state expired | Inspect cookies and local storage after the click; verify domain/path and save the context state. |
| Saved state has no effect | State belongs to another host or the CMP uses a server-side account preference | Create the state on the exact origin and confirm the preference through the site’s own UI. |
| Native dialog freezes the test | A listener was added without accepting or dismissing | Call dialog.accept() or dialog.dismiss() on every event. |
| Page breaks after cookie blocking | Required third-party cookies or embedded components were blocked | Restore normal cookie behavior and handle the consent UI instead. |
| Overlay disappears but tags still fire | Only the DOM node was removed | Use the site’s reject control or a verified state; removing pixels is not a consent signal. |
Or skip the browser setup
If your goal is a clean screenshot or PDF rather than an interactive crawl, ScreenshotNeo makes one HTTP request to capture a URL. Before the capture it accepts the cookie or consent banner like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and the response reports the result in X-Page-Verdict and X-Billed headers.
Use the API documentation at https://screenshotneo.com/docs/ for authentication and all options.
cURL
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);You can also use custom CSS or JavaScript, hide selectors, wait for a selector, delay, or network idle, choose a device and viewport, load lazy images in full-page captures, block selected requests, set headers, cookies, user agent, timezone, geolocation, or authorization, and capture an element, image, or PDF. An MCP server exposes take_screenshot, get_page_info, and capture_pdf to Claude, Cursor, and other MCP clients. Those controls are useful when a target uses a banner that is not among the known cleanup rules, but they do not turn an image capture into a consent decision for a data-collection crawler.
Every feature is included on every plan. Current monthly plans are:
| Plan | Price | Included shots |
|---|---|---|
| Free | $0 | 1,000 per month; no card |
| Starter | $5 | 3,000 |
| Growth | $15 | 15,000 |
| Pro | $39 | 60,000 |
| Scale | $99 | 250,000 |
| Business | $249 | 1,000,000 |
Yearly billing gives two months free. Start with 1,000 free screenshots a month with no card; paid plans start at $5 for 3,000 shots.
Best Value
Historical context for cookie banners
Older measurements illustrate why behavior varies, but they are not current global estimates. A longitudinal ePrivacy review reported in a 2020 paper found that 49% of sampled websites placed cookies before receiving consent and 28% provided no consent mechanism. In a 2019 UK sampling frame, five CMP vendors—QuantCast, OneTrust, TrustArc, Cookiebot, and Crownpeak—accounted for approximately 58% combined market share in the cited methodology. Treat these figures as historical and study-specific, not as a prediction for the site you are scraping.
FAQ
Can I debug the same flow without headless mode?
Yes. Set Playwright’s headless option to false or remove Selenium’s headless argument, slow the actions, and inspect the banner and frames in a visible browser. Switch back to headless only after the site-specific selectors and waits are stable.
How should I prove which consent action a job took?
Store a structured log containing the target origin, UTC timestamp, selected action, selector or frame used, and whether a prior state was restored. Keep screenshots or HTML only when your data policy permits them, and protect any state file that may contain authentication cookies.
Recommended Free Tools
When should a saved consent state be rebuilt?
Rebuild it after a consent-manager redesign, a domain migration, a browser-profile change, or an expiry observed in the cookie/local-storage record. A state file is an optimization for a verified decision, not a permanent substitute for checking the site’s current controls.
Frequently Asked Questions
Can I debug the same flow without headless mode?
Yes. Set Playwright’s headless option to false or remove Selenium’s headless argument, slow the actions, and inspect the banner and frames in a visible browser. Switch back to headless only after the site-specific selectors and waits are stable.
How should I prove which consent action a job took?
Store a structured log containing the target origin, UTC timestamp, selected action, selector or frame used, and whether a prior state was restored. Keep screenshots or HTML only when your data policy permits them, and protect any state file that may contain authentication cookies.
When should a saved consent state be rebuilt?
Rebuild it after a consent-manager redesign, a domain migration, a browser-profile change, or an expiry observed in the cookie/local-storage record. A state file is an optimization for a verified decision, not a permanent substitute for checking the site’s current controls.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
