For reliable e-commerce data, don’t ask an AI agent to scrape every page with a browser by default. First check whether the product details are already available in structured data or a permitted data-bearing request. Use Playwright when the page’s rendered state or an interaction is necessary, then give the LLM a bounded set of page evidence, require a defined output schema, and validate each result.
What should an AI agent collect from a product page?
Decide what a useful product record means before choosing a scraping method. A practical starting schema includes the product name, brand or identifier, variant, offer price, currency, availability, and product URL. Add evidence fields for important values—such as the exact price and availability text—so a downstream system can check where the result came from.
As an Amazon Associate I earn from qualifying purchases.
Schema.org’s Product and Offer vocabulary is a useful reference for modeling product and offer information. Its price property describes an offer price or price component; the priceCurrency property helps distinguish currencies that might otherwise be confused by a symbol alone. These concepts do not prescribe the only valid schema for an AI agent.
{
"product_name": "string",
"brand": "string|null",
"product_identifier": "string|null",
"variant": "string|null",
"price": "number|null",
"currency": "ISO currency code|null",
"availability": "string|null",
"product_url": "string",
"evidence": {
"price_text": "string|null",
"availability_text": "string|null"
}
}Make fields required or optional deliberately. If a value is absent or ambiguous, instruct the model to return null rather than infer it. Treat different sizes, colors, sellers, and regional offers as separate variants or offers when they have distinct details. A price is useful only if it is associated with the correct selected variant, currency, and availability state.
#1 Best Overall
Choose the lightest source that meets the task
Compare sources by whether they contain the needed fields and state, not by whether they use an AI model or a browser. Scrapy’s guidance favors reproducing the requests carrying the desired data when that is feasible; a browser is useful when rendering or interaction is required or those requests are difficult to reproduce.
| Source or method | Use it when | Strength | Tradeoff |
|---|---|---|---|
| Structured product data, such as JSON-LD | The page publishes the needed product and offer fields. | Machine-readable without interpreting page layout. | It may be absent, incomplete, stale, or inconsistent with the currently selected variant. |
| Data-bearing request | The required data arrives in a permitted request that is practical to reproduce. | Can avoid full browser rendering; Scrapy notes potential savings in parsing time and network transfer. | Request formats are site-specific and can change. |
| Playwright-rendered page | Product content appears after JavaScript runs, an interaction reveals it, or the rendered state itself matters. | Observes browser-rendered content and supports controlled interaction. | Requires more resources and state handling; readiness must be tied to the content needed. |
| LLM-assisted extraction | Visible evidence varies and needs mapping into a common schema. | Can interpret inconsistent presentation and map it to chosen fields. | May omit, misread, or invent values unless its input and output are constrained and checked. |
These methods can be combined. For example, structured data can provide candidate fields while a browser confirms which variant is selected. If sources disagree, record the conflict or retrieve the page again rather than silently preferring one.
Check permission and scope before collecting data
Define the target domain, pages, fields, collection frequency, retention period, and intended use. Review the site’s applicable terms and access rules, and check robots.txt for each relevant origin. Google’s documentation explains that robots rules are scoped to a host, protocol, and port, so a rule for one origin should not be assumed to cover another.
Recommended Free Tools
Robots rules describe crawler access; they do not, by themselves, settle whether an agent may submit forms, make purchases, or use collected data. Public visibility, a successful request, or a working browser interaction is not permission to collect or reuse information. Whether a particular activity is permitted depends on the target site, jurisdiction, data, access method, and downstream use; a generic workflow cannot resolve those specifics.
The Agent-to-Web Framework describes a proposed siteai.json file for expressing actions that agents may perform, distinct from URL crawl rules in robots.txt. The proposal is labeled work in progress, not an established universal requirement or settled legal standard.
Inspect the page before launching a browser for every product
- Check the page’s source and structured data. Look for product and offer details, and whether they correspond to the page’s current variant and region.
- Inspect the page’s network activity. Determine whether the needed information arrives through a request. If a permitted, stable request is feasible to reproduce, it may be more efficient than rendering each product page.
- Use Playwright when the rendered page matters. Choose it for client-rendered content, interactions that reveal information, or browser state that the other sources do not adequately provide.
- Stop if access is blocked or challenged. Do not attempt to evade access controls, CAPTCHAs, login restrictions, or rate limits. Use an authorized route if the site blocks the workflow.
Use Playwright to wait for the product state you need
A page’s load event does not prove that its product content is ready. Modern sites can continue fetching data and populating the interface afterward. Playwright’s navigation guidance notes that there is no single moment when every page is “loaded”; readiness depends on the page and framework.
Wait for a meaningful condition in the task, such as the product title, the selected variant, a price, or an availability state. Playwright actions wait for their target elements to become actionable, but the workflow still needs to identify the right target and state. A fixed sleep is fragile: the same delay can be excessive on one visit and too short on another.
For an agent, limit the context to the relevant page region and fields instead of giving it an unrestricted page and an open-ended instruction. Playwright’s ariaSnapshot supports an AI-oriented mode with element references and iframe snapshots. Such a snapshot can help provide bounded page context; it does not establish that every value is present, complete, or correct.
Constrain the LLM to evidence-backed extraction
Give the model the schema, the task, and the evidence it may use. Ask for only the requested fields. Require null for missing or ambiguous values, and preserve the page’s original text for important values rather than asking the model to reconstruct it from memory.
- Keep variant identity alongside its price and availability.
- Do not treat a currency symbol alone as an unambiguous currency code.
- Do not merge offers from different sellers or regions into one record.
- Keep the product URL tied to the page from which the evidence was retrieved.
- When the evidence does not support a field, return the agreed unknown value rather than filling the gap with a guess.
An LLM can map varied page presentations into a shared structure, but that is not proof of accuracy. Treat every extracted value as a candidate until it passes validation.
Validate records and preserve their provenance
Parse the response against a strict schema before storing or passing it to an agent. Check types, required fields, currency format, URL identity, and whether each important value can be traced to page evidence. Keep the retrieval time, page URL, selected variant, and the evidence text or structured-data path used for key fields.
When structured data, visible text, and a network response are all available, compare them rather than assuming one is infallible. Record disagreements or retrieve the page again; do not choose a value silently. To assess quality, repeatedly extract a sample, compare records with a human-checked set, track missing fields, and measure disagreements between sources. These are practical checks, not guarantees of a particular accuracy rate.
Keep collection bounded and respectful
- Set explicit page and request budgets and conservative concurrency.
- Back off after transient errors rather than retrying aggressively.
- Identify the client appropriately where applicable.
- Cache responses where permitted and avoid unnecessary repeat retrievals.
- Stop on access-denied responses, bot challenges, or unexpected authentication requirements.
These operational safeguards do not replace checking the specific site’s terms, applicable law, or data-use restrictions.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
