Recommended Free Tools
Build a product scraper in two stages: collect product URLs and summary fields from category or search-result pages, then visit each product page for richer details. Follow the site’s actual pagination, choose selectors based on the HTML you receive, and investigate the data request when a field appears in a browser but not in the response. First confirm that your target site and intended use permit the crawl; technical accessibility alone does not establish permission.
Plan the crawl and check permission
Before writing a spider, define the target domain, the fields you need, your purpose, and how often the data must be refreshed. Review the target site’s current terms and crawl guidance, and keep the crawl within the permitted scope. Whether a particular collection or use is allowed depends on the site, jurisdiction, and intended use; there is no blanket permission established by the technical steps below.
As an Amazon Associate I earn from qualifying purchases.
Start with a small, explicitly chosen set of category or search-result URLs. Decide which fields belong in the listing record—such as a product name, summary price, and product URL—and which require the detail page. Keeping those page types separate makes the crawler easier to validate and change.
Inspect listing and detail pages
Choose a representative listing page and product detail page. Compare the original HTTP response HTML with what the browser displays. Identify the repeated product-card structure, stable product links or identifiers, pagination control, and the location of each field you want to extract. Exact selectors differ by site, so inspect the target rather than copying selectors from another store.
#1 Best Overall
Scrapy selectors use CSS or XPath against responses and integrate with Parsel/lxml. Use a selector that targets the repeated card, then extract fields relative to that card to avoid accidentally mixing values between products. See Scrapy’s selector documentation.
Discover product URLs
Use the selected listing URLs as crawl entry points, or use a sitemap as an additional URL-discovery input. Scrapy’s SitemapSpider can read sitemap URLs and discover sitemap locations through robots.txt; URL patterns can route product and category pages to different callbacks. A sitemap supplies candidate URLs, not permission for every crawl or use. See Scrapy’s spider documentation.
Parse listings and follow pagination
For each listing response, extract one record per product card. Resolve relative product links against the response URL so the stored URL is absolute. Extract summary fields only when present, and follow the page’s real “next” link rather than assuming the first page contains the whole catalog. Scrapy’s tutorial demonstrates extracting items and yielding a request for the next page: Scrapy tutorial.
- Parse each card. Extract the product URL and the summary fields your project needs.
- Queue detail requests. Yield a request for each discovered product URL and route it to a separate detail callback.
- Queue the next listing page. Resolve the next-link URL against the current response and yield a listing request.
- Stop safely. Stop when there is no next link. Track visited listing URLs or otherwise guard against repeated URLs and pagination loops.
Keep a stable product URL or identifier so records from listings and detail pages can be joined and duplicates detected.
Extract product details separately
In the detail callback, extract only fields that are present and relevant to your use. Depending on the page, these may include product name, brand, SKU, description, price, availability, and variant choices. Store a missing field as missing; do not infer it from a neighboring product or silently substitute a listing value for a detail value.
Keep the listing and detail data linked by a stable URL or identifier. This lets you distinguish a product discovered on a listing from one whose detail page could not be fetched or parsed, rather than treating an incomplete record as a complete product.
Choose between HTML, data requests, and browser rendering
Start by checking whether the field is present in the downloaded HTML. If it is, CSS or XPath parsing is usually the direct approach. If a browser shows the field but the response does not, inspect the page source and the browser’s network requests to see whether the value is embedded in JavaScript or returned by another request.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problems- Parse the HTML response when the required values are already in it. This avoids adding a browser-rendering step.
- Reproduce the data request when an identifiable request returns the needed structured data and its URL, method, headers, body, or form parameters can be matched. Scrapy recommends locating the data source and extracting from it when dynamic content is absent from the initial response: Scrapy guidance on dynamic content.
- Use a headless browser when reproducing the request is impractical or the required state exists only after rendering or interaction. Browser automation adds setup and resource use, so use it when the rendered DOM or an interaction is genuinely needed.
Validate and store the results
Check both coverage and correctness before relying on the output. Scrapy callbacks can yield items and more requests; items can then be handled through pipelines or feed exports. See Scrapy’s spider documentation.
- Compare the number of parsed products on representative listing pages with the visible cards.
- Check that product URLs are absolute and unique, and that pagination does not revisit pages.
- Inspect representative extracted values from both listing and detail pages.
- Confirm that detail records correspond to products discovered in listings and that missing fields remain identifiable.
- Use an explicit output structure; retain raw URLs and timestamps when they are useful for the project.
Troubleshooting common failures
A field is visible in the browser but missing from the response
Inspect the original response and the browser’s network requests. The value may be embedded in script data or supplied by a separate request. Prefer extracting from that source when it can be reproduced; use a rendered browser when the needed state depends on rendering or interaction.
The crawler finds products on only the first page
Check that the listing parser extracts the actual next-page link and that the spider yields a request to it. Resolve relative links against the response URL and stop only when the next link is absent. Add a visited-URL guard to catch loops.
Selectors return empty or mismatched values
Verify the selector against the response HTML, not only the browser’s rendered view. Target each repeated card first, then extract fields within that card. Recheck the page structure when the site changes; selectors are specific to the target’s markup.
Detail records cannot be joined to listing records
Use a stable product URL or identifier across both callbacks. Normalize resolved links consistently, and avoid relying on product names alone when names can be missing or shared.
Best Value
Or skip the browser setup
If you need a screenshot of a product listing or detail page rather than a structured catalog crawl, ScreenshotNeo is a website screenshot API and MCP server. Its one-call API can return an image or PDF; it does not replace a crawler that extracts product fields and follows pagination.
For example, request a screenshot of a product page with cURL (see the ScreenshotNeo API documentation):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
ScreenshotNeo accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and response headers indicate the page verdict and billing status. Its MCP server provides take_screenshot, get_page_info, and capture_pdf for AI agents. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots.
Sign up for 1,000 free screenshots a month, with no card required.
Frequently Asked Questions
Can a sitemap replace listing-page pagination?
A sitemap can supply candidate URLs, but it does not establish permission for a particular crawl or intended use.
Should I use a screenshot API to extract a whole product catalog?
No. A screenshot is an image or PDF, not structured product records; use a crawler for fields and pagination. ScreenshotNeo can capture individual pages when a visual record is what you need.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
