An HTTP 200 means the request succeeded at the HTTP layer; it does not mean your scraper received the records it expected or extracted them successfully. First inspect the exact response body and final URL your scraper saw. Then check whether the data is in that response and whether your parser and selector match it.
What HTTP 200 does—and doesn’t—tell you
HTTP 200 OK indicates that a request succeeded. For a GET request, the response includes a representation of the requested resource. It does not certify that the body contains a particular table, list, or set of records your code wants.
As an Amazon Associate I earn from qualifying purchases.
Extraction is a separate step. Your scraper can receive a successful response and still return no rows because the relevant data is absent, the response uses a different format than your parser expects, or your selector does not match the received content. In Python Requests, Response.ok is broader than an exact 200 check: it is true for status codes below 400, not proof of successful extraction.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Start with the response your scraper actually received
Before changing selectors or adding a browser automation tool, capture the response as seen by the scraper. Record its status, final URL after redirects, relevant headers—especially Content-Type—and save the body locally or inspect a safe sample.
#1 Best Overall
Check whether the body is the expected page or instead an intermediate page, login screen, challenge, error page, shell, or empty response. A status code alone cannot distinguish these cases. Scrapy recommends examining the response received by the crawler and comparing it with another HTTP client when necessary: Scrapy’s guide to dynamically loaded content.
Compare the scraper response with the browser
If the browser shows records that are missing from the saved response, the page may obtain them through JavaScript, embedded data, or a separate request. The browser’s visible page is not necessarily the same content returned by the initial page request.
- Open the page in a browser and use its developer tools to inspect network requests.
- Find the request whose response contains the records, then inspect its URL, method, parameters, headers, and response format.
- Reproduce the relevant request in your scraper, including the request sequence or session state only where inspection shows it is needed.
- If the data is impractical to retrieve directly, consider browser rendering as an alternative.
Do not make arbitrary header changes and assume they are a fix. The goal is to reproduce the request that supplies the data, not to guess at what the site might require. Scrapy notes that reproducing a browser request may involve its method, URL, body, headers, and form parameters.
Match the parser to the response format
Once you have located the response containing the data, use an extraction method suited to that content. The correct approach depends on what the response actually contains:
- HTML or XML: use CSS or XPath selectors against the markup.
- JSON: decode the JSON and read the relevant keys; if a JSON field contains HTML, parse that embedded markup separately.
- JavaScript: inspect scripts for embedded data or identify the separate request that returns it.
- PDFs or images: use text extraction or OCR where appropriate rather than treating the response as ordinary page HTML.
Scrapy describes these as distinct cases because a selector for page markup will not automatically extract data from every response format: dynamically loaded content and response formats.
If the data is in the body, test the selector directly
If the saved response contains the target records, run your selector against that exact response rather than the browser-rendered page. Inspect the matches before extracting fields. In Scrapy, .getall() shows all matched values, while .get() returns the first value or None; an empty match list points toward a selector mismatch.
Also distinguish between a missing element and an element that exists but has no text. A selector can match a container successfully while a text extraction step returns nothing. Scrapy’s shell is designed for inspecting a response and trying selectors interactively: Scrapy Shell documentation.
Free tools Windows power users keep installed
One-click scans. No signup required.
What you need to identify the specific cause
“HTTP 200 and zero rows” describes the symptom, not its cause. Without the URL, request code, response body, content type, and selector, there is not enough evidence to say whether the issue is a different request, dynamic rendering, a changed selector, another response format, or a site-specific condition. Capture those details first; they determine which branch of the diagnosis applies.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
