The reliable way to extract website data is to choose the least complicated source that actually contains it: use an official API or feed when available, parse the initial HTML with CSS or XPath selectors when the fields are in the response, reproduce a page’s underlying data request when content is loaded dynamically, and use a headless browser only when request-level extraction is impractical or the rendered page itself is required.
Before writing a scraper, define the fields, pages, update frequency, and access permission you need. That prevents a fragile crawler from collecting far more than your project requires.
Start with the data source, not the scraper
Website extraction projects usually fall into four categories. The same page can move between categories as its implementation changes, so inspect a real response before selecting tools.
| Where the data is | Best first approach | Why |
|---|---|---|
| Official API, feed, or downloadable dataset | Use that supported interface | It normally provides stable fields, documented access, and less parsing. |
| Initial HTML response | HTTP client plus CSS or XPath selectors | The values are already present without running a browser. |
| Separate request made after page load | Inspect the browser network panel and reproduce the request | A JSON or other structured response is usually smaller and easier to validate than rendered markup. |
| Content available only after rendering or interaction | Headless browser automation | The browser can execute JavaScript, wait for elements, click controls, and read the final DOM. |
Define the extraction contract
Write down each field, its expected type, the pages that contain it, and whether the job is one-time or recurring. Include a few representative URLs and a rule for missing values. This list becomes your validation checklist and keeps selectors focused.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minute#1 Best Overall
Check for an API or structured source
Search the site’s developer documentation, page source, feeds, downloadable files, and network activity for a supported data interface. An API may require an account, an access key, pagination, or a rate policy; follow those terms instead of treating the web page as an undocumented database.
If the API returns the fields you need, store the response and the request parameters alongside your extracted records. Keep the original source URL or endpoint and retrieval time when freshness or auditability matters.
Inspect the actual HTTP response
A browser can display a complete page while a simple HTTP client receives only a shell containing scripts and placeholders. Fetch one representative URL and search the returned text for a known title, price, table cell, or attribute. If it is absent, do not keep adding selectors: change methods.
Simple inspection in Python
import requests
url = "https://example.com/products/42"
r = requests.get(url, timeout=30, headers={"User-Agent": "data-research/1.0"})
r.raise_for_status()
print(r.status_code, r.headers.get("content-type"))
print("Target phrase present:", "Example product" in r.text)
print(r.text[:500])Use a descriptive user agent, reasonable timeouts, and a restrained request rate. A successful status code does not prove that the desired data is present; check the response body and content type.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Extract fields from HTML with CSS or XPath
When values are in the response, selectors can target elements, text, and attributes. Scrapy selectors support both CSS and XPath; Beautiful Soup and lxml are useful alternatives for smaller scripts.
One-page extraction with Beautiful Soup
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
url = "https://example.com/catalog"
r = requests.get(url, timeout=30, headers={"User-Agent": "catalog-parser/1.0"})
r.raise_for_status()
soup = BeautifulSoup(r.text, "html.parser")
rows = []
for card in soup.select("article.product-card"):
name = card.select_one(".product-name")
price = card.select_one(".price")
link = card.select_one("a")
rows.append({
"name": name.get_text(" ", strip=True) if name else None,
"price": price.get_text(" ", strip=True) if price else None,
"url": urljoin(url, link.get("href")) if link and link.get("href") else None,
})
for row in rows:
print(row)Prefer stable attributes such as semantic classes, data attributes, or a meaningful document structure. Avoid selectors tied to generated class names or a particular visual nesting pattern. Treat absent nodes as normal data-quality cases rather than calling .text on a null object.
CSS versus XPath
- CSS is concise for classes, attributes, descendants, and repeated cards.
- XPath is useful when you need relationships such as “the value in the cell next to this label,” text conditions, or traversal to a parent.
- Test selectors against several pages, including a page with missing or optional fields. A selector that works on one example is not yet a reliable extraction rule.
Scale to many pages with Scrapy
Use a crawler framework when you must follow links, manage concurrency, produce structured items, and write through an output pipeline. Scrapy spiders define start URLs and callbacks; callbacks yield dictionaries or item objects, while pipelines can clean and persist them.
A complete minimal spider
import scrapy
class ProductSpider(scrapy.Spider):
name = "products"
allowed_domains = ["example.com"]
start_urls = ["https://example.com/catalog"]
custom_settings = {
"ROBOTSTXT_OBEY": True,
"FEEDS": {"products.jsonl": {"format": "jsonlines", "encoding": "utf8"}},
}
def parse(self, response):
for card in response.css("article.product-card"):
yield {
"name": card.css(".product-name::text").get(default="").strip() or None,
"price": card.css(".price::text").get(default="").strip() or None,
"url": response.urljoin(card.css("a::attr(href)").get()),
"source_url": response.url,
}
next_url = response.css("a.next::attr(href)").get()
if next_url:
yield response.follow(next_url, callback=self.parse)Run it from a Scrapy project with scrapy crawl products. Add detail-page requests only when the fields are not present in the listing. Keep pagination and link discovery constrained to permitted paths so a broad site does not become an accidental crawl.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Robots rules and permission
Read the target site’s robots.txt, terms, and any access instructions. RFC 9309 (September 2022) states: “These rules are not a form of access authorization.” A robots file expresses crawler requests; it does not grant permission to access restricted material. Do not bypass authentication, CAPTCHAs, technical controls, or an explicit prohibition. Scrapy’s ROBOTSTXT_OBEY setting enables its robots middleware.
Extract data from JavaScript websites
If the initial response lacks the values, open browser developer tools and watch the Network panel while the page loads or while you trigger the relevant action. Identify the request that returns the data, including its method, query or JSON body, pagination parameters, and required headers.
Reproduce the underlying request when practical
A direct data request is generally faster, cheaper, and easier to validate than rendering every page. Copy the request as cURL from developer tools, remove browser-only noise, then implement it with an HTTP client. Preserve only headers and cookies that the endpoint actually requires, and respect authentication and usage terms.
import requests
api_url = "https://example.com/api/products"
params = {"page": 1, "limit": 50}
r = requests.get(api_url, params=params, timeout=30)
r.raise_for_status()
data = r.json()
for item in data.get("items", []):
print(item.get("id"), item.get("name"))Check whether the response is paginated, whether a token expires, and whether the endpoint returns a different schema for errors. Save a sample response and validate required keys before scheduling the job.
When to use a headless browser
Use browser automation when the data request cannot be reproduced reliably, when JavaScript computes the values locally, or when the rendered DOM is itself the required output. A browser can wait for a selector, perform clicks, and access content after hydration. Scrapy’s dynamic-content guidance identifies Playwright as an example; for tighter Scrapy integration, use an integration such as scrapy-playwright rather than bypassing Scrapy’s scheduling and item flow with an unrelated browser script.
Browser extraction costs more CPU and memory and introduces timing, cookie, viewport, and browser-version variables. Wait for a meaningful selector or network-idle condition instead of using an arbitrary long sleep, and capture diagnostics when a wait times out.
Validate, normalize, and store the result
- Required fields: reject or quarantine records missing identifiers or essential values.
- Types and encoding: normalize whitespace, Unicode, dates, currencies, and decimal separators before analysis.
- Duplicates: choose a stable key such as a source ID plus URL; do not assume position in a list is stable.
- Coverage: compare discovered pages with expected pagination or link counts.
- Provenance: retain the source URL and retrieval timestamp when records can change.
- Representative review: inspect successful, empty, changed-layout, and error responses before trusting an export.
Write raw responses or a content hash when reproducibility matters, then store cleaned records separately. This lets you distinguish a source change from a parser bug.
Choose the method by project shape
| Project condition | Preferred method | Main trade-off |
|---|---|---|
| Supported API | API client | Requires learning authentication, quotas, and schema. |
| One or a few static pages | Requests plus Beautiful Soup or lxml | Selectors must be maintained when markup changes. |
| Many linked pages | Scrapy | More project structure and crawl controls to configure. |
| Data endpoint visible in Network tools | Reproduced HTTP request | Tokens, signatures, or undocumented changes can break it. |
| Rendered-only content or interactions | Headless browser | Highest operational complexity and resource use. |
Common failures and fixes
The response is an HTML shell
Cause: JavaScript loads the records later. Fix: inspect Network requests for a JSON or other data response; reproduce it, or use a browser if no stable request-level route exists.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →A selector returns no values
Cause: the selector targets generated classes, a different template, or the wrong document. Fix: save the response, inspect its structure, test a simpler selector, and add a template-specific branch only when the difference is real.
403, 429, or repeated timeouts
Cause: access controls, rate limits, overloaded pages, or an incorrect request. Fix: stop increasing concurrency, verify permission and required headers, use backoff and bounded retries, reduce scope, and stop if the site indicates automated access is unwanted. Never attempt to bypass a CAPTCHA or authentication wall.
JSON parsing fails
Cause: the server returned an HTML error page, a login page, or malformed data. Fix: inspect status, content type, and the first response bytes before calling .json(); log the request URL without exposing secrets.
Browser waits forever
Cause: the chosen selector never appears, the page is blocked, or a request remains open. Fix: verify the selector manually, wait for the smallest meaningful element, set a finite timeout, record a screenshot and console/network errors, and classify the page as failed rather than emitting an empty record.
Records suddenly become empty
Cause: a layout or schema change. Fix: keep sample responses, monitor required-field counts, alert on unusual zero or null rates, and update selectors only after inspecting the new source.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Or skip the browser setup
For a screenshot rather than structured field extraction, ScreenshotNeo provides a one-request website screenshot API. It accepts consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be disabled. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and the response identifies the result with X-Page-Verdict and X-Billed headers. Its MCP server provides take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.
See the ScreenshotNeo documentation for all options, including full-page and element capture, lazy-image loading, device presets, retina scale, PDF settings, custom CSS and JavaScript, clicks, waits, request blocking, cookies, headers, user agents, timezone, geolocation, transparent backgrounds, resizing, TTL caching, signed links, asynchronous webhooks, bulk capture, usage data, and the OpenAPI specification.
cURL
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);ScreenshotNeo’s Free plan includes 1,000 screenshots per month with no card. Paid plans start at $5 for 3,000 shots; yearly billing gives two months free, and every feature is included on every plan. Create a free ScreenshotNeo account.
Recommended Free Tools
FAQ
Is web scraping the same as extracting data?
Scraping is one way to extract data, usually by requesting and parsing pages at scale. Extraction can also use an official API, feed, dataset, or a single manually selected page.
Best Value
Should I save HTML or JSON?
Save the raw form that best explains the source you used. Keeping a response or content hash alongside cleaned records helps you reproduce a result and diagnose later changes.
How do I know whether a page is dynamic?
Compare a browser view with the raw HTTP response. If the desired text is missing from the response but appears after load, inspect the browser’s Network panel for the request that supplies it.
Can robots.txt make scraping legal?
No. It communicates crawler preferences and is not access authorization. Permission, terms, authentication boundaries, and applicable law remain separate questions.
Frequently Asked Questions
What is the safest first step before scraping a site?
Check for an official API, feed, or downloadable dataset, then confirm the site’s terms and access requirements before requesting pages.
What should I do when a website changes its layout?
Keep representative raw responses and monitor required-field and null rates; inspect the new markup or schema before changing selectors.
When is a screenshot service preferable to data extraction?
Use a screenshot service when the deliverable is the rendered visual or PDF, not a normalized set of fields.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
