Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteShort answer: Puppeteer is officially a JavaScript library, not a Python package. Python developers can use the unofficial pyppeteer port, but its own repository says it is unmaintained. More importantly, Google says automated Search queries and scraping results without express permission violate its spam policies and Terms of Service. Use browser automation only on pages and data you are authorized to access; for a merchant’s catalog, Google’s supported product-data and structured-data options are usually the more stable route.
This guide shows the architecture, a controlled Python example, deployment considerations, failure modes and safer alternatives. It intentionally does not publish supposedly “working” Google Shopping selectors: Google’s DOM is changeable, and the available official documentation does not establish a stable extraction interface.
What “Puppeteer with Python” actually means
Chrome for Developers documents Puppeteer as a JavaScript library that automates Chrome and Firefox through the Chrome DevTools Protocol and WebDriver BiDi. It can query DOM elements, click, type, and intercept or modify network requests and responses. Those capabilities describe browser automation generally; they do not guarantee a supported or stable way to extract Google Shopping result pages.
pyppeteer is an unofficial Python port. Its repository says the project is unmaintained, requires Python 3.8 or later, and may download Chromium on first use when a suitable browser is unavailable. Treat those details as version-sensitive: check the repository and your installed browser before deploying.
Recommended Free Tools
#1 Best Overall
| Path | Language | Maintenance and compatibility | When it fits |
|---|---|---|---|
| Official Puppeteer | JavaScript/Node.js | Official project documentation; current browser support follows the project’s releases | Teams able to run Node.js and wanting the primary API |
| pyppeteer | Python | Unofficial port; repository states it is unmaintained | Legacy or tightly controlled experiments where Python is required and compatibility is tested |
| Merchant-owned product data | Any stack | First-party, structured and intended for Google’s ecommerce systems | Catalog owners who need durable product publishing or analysis |
If you can choose the runtime, official Puppeteer in Node.js is the less ambiguous implementation. If Python is mandatory, isolate pyppeteer behind a small service so replacing it later does not affect the rest of your application.
Check authorization and choose the right data source
Google Search Central’s machine-generated traffic policy says automated queries and scraping Search results without express permission violate Google’s spam policies and Terms of Service. Do not bypass CAPTCHA or bot checks, rotate identities to evade controls, disguise automation, or scale an unapproved collector. This is a policy statement from Google, not a legal opinion about every jurisdiction.
For a retailer that owns the catalog, start with Google’s ecommerce guidance: use supported product-data sharing methods and structured data so Google can understand and present your products. That gives you authoritative records instead of attempting to parse a consumer-facing page.
Google’s crawling documentation explains that Storebot-Google crawl preferences affect Google’s own crawling of Shopping surfaces. They do not grant a third party permission to scrape Google result pages.
A controlled Python browser-automation example
The following example is deliberately pointed at a page you own or have written permission to test. It demonstrates navigation, waiting, reading visible text and saving HTML without claiming that any selector is valid for Google Shopping.
1. Create an isolated environment
- Install Python 3.8 or newer, then create and activate a virtual environment:
python -m venv .venvsource .venv/bin/activate(Linux/macOS) or.venvScriptsactivate(Windows). - Install the unofficial port:
pip install pyppeteer - On first launch, pyppeteer may download Chromium. In a build pipeline, pin and cache dependencies, and verify that the downloaded browser is compatible with your pyppeteer version.
2. Run the script
Save this as authorized_capture.py and replace the URL and selectors with ones documented for your own test page.
import asyncio
from pathlib import Path
from pyppeteer import launch
TARGET_URL = "https://example.com/catalog"
ITEM_SELECTOR = "[data-test='product-card']"
async def main():
browser = await launch(
headless=True,
args=["--no-sandbox", "--disable-setuid-sandbox"],
)
page = await browser.newPage()
await page.setViewport({"width": 1440, "height": 1000, "deviceScaleFactor": 1})
await page.goto(TARGET_URL, {"waitUntil": "networkidle2", "timeout": 60000})
await page.waitForSelector(ITEM_SELECTOR, {"timeout": 15000})
rows = await page.evaluate("""(selector) => Array.from(document.querySelectorAll(selector)).map(card => ({
name: card.querySelector('[data-test=product-name]')?.textContent?.trim() || null,
price: card.querySelector('[data-test=product-price]')?.textContent?.trim() || null,
link: card.querySelector('a')?.href || null
}))""", ITEM_SELECTOR)
Path("catalog.json").write_text(
__import__("json").dumps(rows, indent=2), encoding="utf-8"
)
await page.screenshot({"path": "catalog.png", "fullPage": True})
await browser.close()
if __name__ == "__main__":
asyncio.get_event_loop().run_until_complete(main())The important design choice is that selectors are owned by the page operator and covered by tests. A selector such as a class name copied from a public result page can disappear without notice. Prefer stable data-test attributes on pages you control, and store a fixture HTML page for regression tests.
3. Add bounded retries and evidence
Use a finite retry count with increasing delays, record the final URL and HTTP-visible outcome, and save a screenshot or HTML snapshot when parsing fails. Never turn retries into an attempt to defeat a block. Set a navigation timeout, cap the number of pages, and close every browser in a finally path in production code.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Why a Google Shopping scraper breaks
Selectors and markup change
There is no selector, pagination scheme or result-count contract established by the cited official materials. Keep extraction code behind an adapter, test it against fixtures, and expect maintenance whenever the page structure changes.
Consent, login and bot-interstitial pages
A consent screen, sign-in wall, CAPTCHA or other interstitial means the expected product elements are not present. Detect the page state, stop, and follow the site owner’s approved access method. Do not add stealth techniques or CAPTCHA-solving.
Timeouts and incomplete rendering
Network-idle is not proof that every lazy element is ready. Wait for a page-owned readiness marker, use a reasonable timeout, and distinguish an empty authorized page from a failed load in your logs.
Chromium incompatibility
Because pyppeteer is unmaintained, a newer browser or Python release can expose compatibility problems. Pin versions in a lockfile, run a smoke test after upgrades, and consider moving to official Puppeteer if JavaScript is acceptable.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Performance, reliability and deployment
One browser process per job is simpler but expensive. Reuse a browser for a bounded batch, create a fresh page per task, and limit concurrency to what your authorized origin can handle. Cache only data you are permitted to retain, use backoff for transient failures, and make jobs idempotent so a retry does not duplicate records.
Google Cloud’s Cloud Run browser-automation documentation describes installing Chromium and using high-level libraries such as Puppeteer or Playwright, as well as the Chrome DevTools Protocol. Cloud Run can host an authorized browser workload; hosting does not change Google Search’s access policy.
- Set explicit CPU, memory and timeout limits.
- Keep Chromium and the Python package aligned and test cold starts.
- Send structured logs for URL, status, duration, parser version and failure category.
- Keep credentials, cookies and authorization headers in a secret manager, never in source code.
- Use a queue for scheduled work and a dead-letter path for repeated failures.
Safer alternatives for catalog and price data
If you own the products, publish feeds and structured data through Google’s supported ecommerce methods instead of rebuilding a consumer-result scraper. If you have a contractual data provider, consume its API or export and retain the provider’s usage terms. For market research, obtain express permission and define fields, rate limits, retention and deletion rules before automating.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Or skip the browser setup
For an authorized page where you simply need a rendered image or PDF, ScreenshotNeo makes one GET request and returns PNG, JPEG, WebP or PDF. It accepts consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups and chat widgets before capture; each step can be disabled. Bot checks, blank pages, timeouts, failed loads and cache hits are not billed, and response headers identify the page verdict and billing result. Its MCP server exposes take_screenshot, get_page_info and capture_pdf to Claude, Cursor and other MCP clients.
cURL (see the ScreenshotNeo documentation):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);ScreenshotNeo includes full-page capture, lazy-image loading, CSS-selector element capture, dark mode, device presets, custom viewports, retina scale, PDF controls, custom CSS and JavaScript, clicks, waits, request blocking, headers, cookies, user agents, authorization, timezone, geolocation, transparent backgrounds, resizing, configurable caching, signed links, asynchronous webhooks, bulk capture for up to 100 URLs per call, usage APIs and an OpenAPI specification. Parameter names used by other screenshot APIs also work for easier migration.
The Free plan includes 1,000 screenshots each month with no card. Paid plans start at $5 for 3,000 shots; all features are available on every plan, and yearly billing gives two months free. Start with the free ScreenshotNeo account.
Troubleshooting checklist
- “Browser executable not found”: allow the first-run Chromium download or configure a tested local executable path; verify the package/browser pair.
- “Timeout waiting for selector”: confirm the authorized page URL, readiness condition and selector; save HTML and a screenshot to see whether an interstitial appeared.
- Empty results: check whether content is rendered after interaction, whether your account has access, and whether the page has changed. Do not infer that an empty list means zero products.
- Process crashes: lower concurrency, close pages in
finally, and raise memory limits only after measuring the workload. - Repeated blocks: stop the job and contact the site owner or use an approved feed/API. Changing user agents or proxies to evade controls is not a fix.
Frequently Asked Questions
Can I use official Puppeteer directly from Python?
No. Official Puppeteer is a JavaScript library. Python projects commonly encounter pyppeteer, an unofficial port whose repository says it is unmaintained.
Does Storebot-Google permission let me scrape Shopping results?
No. Storebot-Google preferences describe Google’s crawler behavior on Shopping surfaces; they do not grant third-party permission to collect result pages.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchShould I deploy this on Cloud Run to avoid Google blocks?
No. Cloud Run can host authorized Chromium automation, but deployment location does not change Google’s access rules or authorize scraping.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
