Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesA page can look complete in your browser while a crawler receives only an almost-empty HTML shell. The difference is JavaScript: browsers can run it to reveal content and links, while a basic HTTP crawler sees only the server’s initial response. A reliable crawler checks what that response contains, renders pages that need JavaScript, and follows robots.txt deliberately. The title describes a build, but no implementation details or test results are established here, so this article focuses on the engineering problem and the design choices it demands.
Why a page works in a browser but not in a crawler
A normal browser does more than download HTML. It can also run JavaScript, make additional requests, and update the page after its initial load. In an app-shell design, the initial response may contain little more than the structure and scripts needed to build the visible page. A person sees the finished result; a crawler that only downloads HTML may see no meaningful text or links.
As an Amazon Associate I earn from qualifying purchases.
Google documents this as a two-stage process for its own crawler: it first processes the HTTP response, then queues pages for rendering with headless Chromium. The rendered output can expose content that was not present in the original response. Rendering may happen later, so a page’s appearance in a browser is not proof that its content was available in the initial response or immediately accessible to a crawler. Google Search Central’s JavaScript SEO documentation explains the distinction.
Free tools Windows power users keep installed
One-click scans. No signup required.
How to find out what your crawler actually receives
Start with the crawler’s HTTP response, not a screenshot of the page. Inspect the response body and status code from the same URL your crawler requests. Look for the text and links the crawler needs. If the response contains the content, a plain HTTP fetch may be enough. If it contains only a shell and scripts, the site may require JavaScript execution before that content appears.
#1 Best Overall
This check separates two different questions: whether the server returned the necessary content, and whether a browser can construct it later. It also helps identify whether an apparent extraction failure is really a rendering issue or instead involves the URL, response, or crawler’s parsing.
When a crawler needs a browser-rendering stage
Use browser rendering when the required content or links are absent from the initial response and become available only after the site’s JavaScript runs. A headless browser can execute that code and expose the resulting page, but it costs more in time and resources than a straightforward HTTP fetch.
That cost argues against rendering every URL by default. Apache StormCrawler documents a selective pattern: use a cheaper HTTP fetch to detect pages that are likely JavaScript-rendered, then route those pages to Playwright. This keeps browser work focused on pages that need it rather than making every crawl pay the rendering cost. The Apache StormCrawler 3.x documentation describes the approach.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →| Question | Plain HTTP fetch | Browser-rendered fetch |
|---|---|---|
| Is the required content in the initial response? | Can extract it if present. | Can also expose content built after the response. |
| Must site JavaScript run to reveal content or links? | No; it does not execute the page’s JavaScript. | Yes; it can run JavaScript in a browser engine. |
| Cost and latency | Cheaper and generally faster, as described by Apache StormCrawler. | More expensive; reserve it for pages likely to need rendering. |
| What else must the crawler respect? | HTTP status and applicable robots.txt rules. | HTTP status and applicable robots.txt rules still matter. |
Google recommends server-side rendering or pre-rendering as a way to make content available without relying on every crawler to execute JavaScript. It notes that these approaches can help users and crawlers, and that not all bots can run JavaScript. Google’s JavaScript SEO guidance covers this trade-off.
Rank #3
What robots.txt does—and does not do
robots.txt is a set of crawler instructions, not a security boundary. The Robots Exclusion Protocol, standardized as RFC 9309 by the IETF in September 2022, tells crawlers how to interpret rules published for them. A crawler should fetch and parse the file, apply the rules that match its user-agent and requested path, and handle redirects, errors, and caching according to the standard. RFC 9309 requires a parser limit of at least 500 kibibytes and says crawlers should not use cached robots.txt content for more than 24 hours in ordinary conditions unless the file is unreachable. See RFC 9309.
The standard is explicit: “These rules are not a form of access authorization.” A disallowed URL may still be known or appear in search results if other pages link to it, even if a search crawler does not fetch its contents. Google describes this limitation in its robots.txt introduction. Keep private material behind genuine access controls, such as authentication, rather than relying on a crawl rule.
Rank #4
A practical design for a crawler
- Fetch responsibly. Retrieve the URL over HTTP, record the response status and body, and apply the site’s parseable robots.txt rules. RFC 9309 specifies crawler behavior for successful fetches, redirects, unavailable files, unreachable files, and caching.
- Check for the content you need. Parse the initial response for target text and links. Do not assume a page is empty merely because one extraction rule failed; first determine whether the content exists in the received HTML.
- Render selectively. If the response is only an app shell and the required content appears after JavaScript runs, send that URL to a browser-rendering stage. Keep ordinary pages on the cheaper HTTP path.
- Extract from the rendered result. When rendering is required, collect the target content and links from the page after its scripts have run, while retaining the original HTTP status and crawl-policy checks.
What this build story can—and cannot—claim
The title establishes that a website crawler was built, but it does not identify the language, libraries, particular bugs, websites tested, crawl scale, or measured results. Those details should not be inferred. The documented engineering lesson is narrower and useful on its own: browser-rendered appearance and initial-response HTML are different inputs, and a crawler should distinguish them rather than treating “it works in the browser” as proof that a basic fetch can see the page.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
