Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →When a store redesign breaks a price scraper, first find out what changed: the response your crawler receives, the request that supplies the price, or the rendered page. Use a selector only if the price is in the response being parsed; reproduce the data request when the price comes from an API-like JSON or HTML response; use a headless browser when the value is available only after rendering or the request is impractical to reproduce. Then validate the extracted fields so a crawl that finishes successfully cannot quietly deliver missing or incorrect prices.
Diagnose the response before changing selectors
A page that looks correct in a regular browser may not contain the same data in the response your scraper processes. Start by capturing the response received by the crawler and comparing it with the browser view. Scrapy’s dynamic-content guide describes using its fetch command to save the response as Scrapy sees it.
As an Amazon Associate I earn from qualifying purchases.
Look for the price in the initial HTML, embedded JavaScript, or a separate network request. If an ordinary HTTP client sees data that Scrapy does not, compare request details such as the user agent and headers before assuming a selector is broken. Redirects, server errors, inconsistent responses, or request blocking can also explain missing data; Scrapy notes that intermittent expected responses may point to target-server problems, overload, or banning.
Choose an extraction method that matches the data source
| What you find | First choice | Trade-off |
|---|---|---|
| Price is in the initial response HTML | CSS or XPath selectors against that response | Lightweight and straightforward, but dependent on the value being present and the selector identifying the intended price. |
| Price arrives in a separate JSON or HTML request | Reproduce that request and parse its response | Often provides structured data, but you may need the original method, URL, headers, body, or form parameters. |
| Price appears only in the rendered page, or reproducing its request is impractical | Use a headless browser such as Playwright | Can inspect rendered DOM, with additional browser execution and integration overhead. |
| The crawl runs, but extracted fields may be missing or wrong | Add field and record validation, monitoring, and alerts | Surfaces silent failures; checks must fit the store, product, and price format. |
Scrapy’s guidance on dynamically loaded content says reproducing the requests that contain desired data is the preferred approach when pages fetch data from additional requests. It can provide structured, complete data while reducing parsing and network transfer. If reproducing the request is difficult or the required value exists only in the rendered DOM, Scrapy describes a headless browser as a fallback. For Playwright within a Scrapy crawl, the docs recommend scrapy-playwright integration rather than using Playwright in a way that bypasses Scrapy components such as middleware and duplicate filtering.
#1 Best Overall
Test selectors against the current response
For HTML extraction, use selectors on the response your crawler actually receives, not just on a browser’s live DOM. Scrapy supports CSS and XPath and provides an interactive shell for inspecting responses and trying expressions; see its Selectors documentation.
Check the number of matches as well as the value. In Scrapy, .get() returns the first match, or None if there is no match; .getall() returns all matches. A page may contain a current price, an old price, and prices for other variants, so accepting the first result without checking can produce a plausible but wrong record. Validate that the selector finds the intended field exactly as expected, and treat zero or unexpected multiple matches as a condition to investigate.
Use browser locators carefully
When browser automation is necessary, Playwright recommends locators based on user-facing attributes and explicit contracts such as accessible roles. Its locator guidance can make interactions clearer, but a retailer controls its own markup: a price is not guaranteed to have a unique or meaningful accessible role. Narrow the locator with relevant page or product context, then validate the extracted text and parsed value.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Validate data, not just whether the spider finished
A successful process exit does not prove that the price extraction still works. Scrapy’s extensions page warns that “Spiders fail quietly in production” and describes Spidermon for monitoring, validation, and alerts. Useful checks include:
- Require a price where one is expected; flag missing, unparsable, or unexpectedly duplicated values.
- Check currency and whether the parsed amount is plausible for the product, including against prior observations where appropriate.
- Track the number of products extracted and the share with valid prices; alert on meaningful drops from that crawler’s own baseline.
- Keep the URL, timestamp, and a diagnostic response sample or artifact where permitted, so a failure can be reproduced.
There is no universal threshold for an acceptable drop or price change in the cited documentation. Set alert levels from each crawler’s historical behavior and the store’s pricing patterns, rather than applying an invented one-size-fits-all percentage.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Keep repairs localized and check site-specific constraints
Put page-specific selectors and price transformations in a focused extraction module or page object rather than scattering them through request scheduling and persistence code. Scrapy describes scrapy-poet page objects as a way to separate extraction from parsing, supporting reuse and testing; see its extensions information.
Before implementing a crawler for a particular store, verify that store’s current page structure, available data requests, and applicable terms or permissions. General Scrapy and Playwright documentation does not establish permission to scrape any specific retailer, nor does it establish that a particular store has a public API. The techniques above also have no universal speed or reliability ranking: choose based on data completeness and correctness, maintenance effort, runtime and network cost, and how well failures can be observed.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
