For ordinary multi-page HTML crawling, start with Scrapy. It gives a Python project a scheduler, asynchronous requests, link following, selectors, item pipelines and exports. Choose Crawlee for Python when you want one asyncio-oriented interface for HTTP and browser crawling, persistent queues, retries and storage. Choose Playwright, Selenium or Puppeteer when the target depends on JavaScript or user-like interaction; these are primarily browser-automation tools, not complete crawl-management systems. Scrapy can combine both jobs through its official scrapy-playwright integration.
There is no neutral 2026 benchmark proving one framework is fastest or universally best. The practical choice depends on rendering, crawl scale, language, operational controls and the difference between free framework code and paid infrastructure.
Quick decision: which framework fits your crawl?
| Need | Best starting point | Reason |
|---|---|---|
| Many ordinary HTML pages, structured extraction and feeds | Scrapy | Full crawler workflow with scheduling, callbacks, selectors, pipelines and exports. |
| Python plus HTTP and browser crawling behind one interface | Crawlee for Python | HTTP and Playwright crawlers, retries, persistent request queues, sessions, proxies and storage. |
| JavaScript-rendered content or interaction | Playwright, Selenium or Puppeteer | A real browser can execute JavaScript and perform clicks, typing and navigation. |
| Scrapy workflow with browser rendering | Scrapy plus scrapy-playwright | Browser-loaded HTML is returned to Scrapy callbacks and selectors. |
| One-off local experiment | Requests/BeautifulSoup or a small Crawlee script | Less project structure is needed when there is no queue or long-running crawl. |
The Apify 2026 survey reported that 71.7% of respondents used Python for scraping and 17% preferred JavaScript. It also listed Selenium, Puppeteer, Playwright and Scrapy among the most-used frameworks by its respondents. The survey was shared mainly with the Apify and The Web Scraping Club communities, so treat those figures as self-reported results from that audience, not a global census or a quality ranking.
What makes a framework “free”?
Scrapy, Crawlee and browser-automation packages are free software to install, but a complete crawl can still incur costs. Browser binaries consume CPU and memory; persistent jobs need a machine or hosting; proxies, managed browser rendering, storage and high-volume APIs are separate services. A local learner may pay nothing beyond their computer, while a production crawl may need paid compute or infrastructure. Framework cost therefore is not the same as total operating cost.
Recommended Free Tools
#1 Best Overall
Regardless of tool, follow the target site’s terms, robots guidance and applicable law. Browser rendering does not grant permission to bypass access controls, CAPTCHAs or authentication barriers.
Scrapy: the strongest default for multi-page HTML
What Scrapy provides
Scrapy is an application framework for crawling websites and extracting structured data. Its asynchronous engine schedules requests, invokes spider callbacks, follows links, applies CSS or XPath selectors, and sends extracted items through pipelines. Feed exports can produce JSON, CSV or XML, while storage backends and extensions support longer-running projects. Robots.txt support, request delays, per-domain concurrency limits and AutoThrottle help you make a crawl less aggressive.
Minimal spider
import scrapy
class ProductSpider(scrapy.Spider):
name = "products"
start_urls = ["https://example.com/catalog"]
def parse(self, response):
for card in response.css("article.product"):
yield {
"name": card.css("h2::text").get(),
"price": card.css(".price::text").get(),
"url": response.urljoin(card.css("a::attr(href)").get()),
}
next_url = response.css("a.next::attr(href)").get()
if next_url:
yield response.follow(next_url, callback=self.parse)Create a project with scrapy startproject catalog, put the spider in its spiders directory, then run scrapy crawl products -O products.json. Use scrapy shell https://example.com/catalog to test selectors interactively before running a full crawl.
When plain Scrapy is not enough
A normal HTTP response can contain only an empty HTML shell when a site inserts content after JavaScript executes. The official scrapy-playwright extension runs a real browser and returns the loaded page inside Scrapy’s request, callback and item-pipeline workflow. This keeps Scrapy’s crawl controls while adding browser overhead.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsScrapy trade-offs
- Strengths: mature crawler architecture, asynchronous scheduling, link following, exports, throttling and extensive extension points.
- Costs: browser rendering requires extra setup and resources; a pure HTTP spider cannot see content that never arrives in the initial response.
- Best fit: repeatable crawls over many pages where queueing, extraction and output matter more than visual interaction.
Crawlee for Python: one interface for HTTP and browsers
Why choose it
Crawlee for Python is an open-source library released under the Apache License 2.0. Its project documentation describes automatic parallel crawling, retries, request routing, a persistent request queue, session management, proxy rotation and pluggable data or file storage. You can use a BeautifulSoup-based HTTP crawler for ordinary pages and a Playwright crawler when JavaScript execution is required. It can run anywhere; deploying to Apify is an option, not a requirement.
Conceptual workflow
A Crawlee application typically adds initial URLs to a request queue, lets the crawler fetch each request, routes responses to handlers, retries transient failures and writes records to chosen storage. Select an HTTP crawler when the response already contains the data. Switch to the Playwright crawler for rendered content or interactions without redesigning the whole queue-and-storage model.
Trade-offs
- Strengths: a shared Python/asyncio style across HTTP and browser jobs, persistent queues, retries, sessions, proxies and storage.
- Costs: browser mode still needs browser binaries and compute; the abstraction can be more machinery than a tiny one-off script.
- Best fit: Python teams that expect to mix cheap HTTP requests with selected browser requests and want operational features early.
Playwright, Selenium and Puppeteer: browsers first
Playwright, Selenium and Puppeteer are commonly used to automate real browsers. They are appropriate when a scraper must wait for JavaScript, click controls, enter text, scroll to trigger lazy loading or capture state that an HTTP client cannot obtain. The Apify survey names all three, along with Scrapy, among its most-used frameworks among respondents.
Do not confuse browser automation with an end-to-end crawl framework. A browser script gives you page control; you may need to add your own URL queue, deduplication, retry policy, persistence, extraction schema, rate limits and export code. For a handful of pages, that simplicity is useful. For thousands of URLs, a crawler framework or a browser-enabled crawler usually provides a clearer operational boundary.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Use a browser when these symptoms appear
- The initial response contains a root element but no records.
- Data appears only after a network request triggered by JavaScript.
- A “load more” button, scrolling, tabs or login flow is required.
- Lazy images or other resources must be loaded before extraction.
Use an HTTP crawler first when the data is present in the server response. It is normally lighter, easier to throttle and cheaper to run than opening a browser for every URL.
How to choose by project requirements
Rendering and interaction
Start with direct HTTP parsing. Add a browser only for the routes that need it. Scrapy plus scrapy-playwright and Crawlee’s mixed HTTP/browser approach both support this split. A browser everywhere is convenient but increases startup time, memory use and failure modes.
Queueing and reliability
For a durable crawl, look for persistent request queues, duplicate filtering, retries, session handling and a way to resume after interruption. Scrapy supplies a scheduler and extensive settings; Crawlee explicitly supplies a persistent queue, retries and session management. A bare browser script requires you to design these pieces.
Language and integration
Python was the preferred scraping language for 71.7% of Apify’s surveyed respondents, while 17% preferred JavaScript. Choose the language already used by your extraction, data-validation and deployment code rather than treating survey popularity as a technical verdict.
Scale and politeness
Set per-domain concurrency and delays, enable AutoThrottle where appropriate, and bound retries. Keep a clear user agent and identify your project when suitable. Persistent queues help resume work, but they do not make unlimited request rates acceptable.
A practical selection procedure
- Inspect one response. Fetch the URL without a browser and check whether the required fields are present in the HTML.
- Choose the lightest fetcher. Use Scrapy or Crawlee’s HTTP crawler for server-rendered content.
- Prove the rendering need. If fields are absent, reproduce the page in a browser and identify the exact wait, click or scroll that reveals them.
- Keep browser scope narrow. Route only JavaScript-dependent URLs through Playwright, Selenium, Puppeteer or a browser-enabled crawler.
- Add operational controls. Configure concurrency, delays, retries, deduplication, persistence, logging and an export format before scaling.
- Measure your own workload. Record success rate, memory, browser share, retries and end-to-end cost. Existing survey usage is not a controlled speed comparison.
Performance, reliability and cost notes
No independent 2026 cross-framework benchmark establishes a speed winner. Network latency, page complexity, concurrency, selectors, proxy quality and browser share dominate results. Compare tools on a representative sample of your own URLs, using the same concurrency and retry policy.
HTTP crawlers usually use fewer resources per request. Browser crawlers can handle rendered interfaces but consume more CPU and memory and introduce browser crashes, navigation timeouts and selector changes. Persistent queues reduce lost work after restarts; retries should be bounded so a permanently broken URL does not consume the entire run. Cache stable responses where permitted, and separate transient network errors from valid empty results in your logs.
Free framework code does not include a universal allowance of hosted runs, proxies or browser minutes. If you move to managed deployment or a managed browser-rendering API, price that service separately from the package license.
Troubleshooting common failures
Empty fields from a page that looks populated
Cause: JavaScript fills the DOM after the HTTP response. Fix: inspect the response source; if the data is absent, use scrapy-playwright, Crawlee’s Playwright crawler or another browser tool, and wait for the relevant selector rather than an arbitrary long delay.
Too many timeouts
Cause: overloaded targets, slow resources, an overly short timeout or a browser waiting for an event that never occurs. Fix: set explicit navigation and selector timeouts, block unnecessary resources where appropriate, limit concurrency, and record the URL and stage that timed out.
Duplicate or missing pages after a restart
Cause: an in-memory queue or unstable URL normalization. Fix: use Scrapy’s scheduler or Crawlee’s persistent request queue, normalize URLs consistently, and persist item output incrementally.
Blocked requests or bot checks
Cause: site controls, request volume, reputation or an access policy. Fix: respect the site’s rules, reduce concurrency, identify your crawler and obtain permission or an approved data feed. Do not design a workflow to defeat a CAPTCHA or access control.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Browser works locally but fails in deployment
Cause: missing browser binaries, sandbox restrictions, fonts, certificates or environment variables. Fix: install the browser and its dependencies in the deployment image, pin compatible versions, log browser and framework versions, and test a single URL in the same environment.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.When a screenshot is the actual requirement
If the output is a visual record rather than extracted fields, a screenshot API can be simpler than maintaining browser infrastructure. ScreenshotNeo is the first option to try: it removes consent banners, newsletter popups and chat widgets before capture, bills only clean shots, and has the lowest paid plan described here.
Or skip the browser setup
One GET request returns a PNG, JPEG, WebP or PDF. See the ScreenshotNeo documentation for parameters.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpimport requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);ScreenshotNeo supports full-page and element captures, dark mode, device presets, retina scale, PDF page settings, custom CSS and JavaScript, clicks, selector waits, network-idle waits, request blocking, headers, cookies, user agents, authorization, timezone, geolocation, transparent backgrounds, resizing, chosen cache TTLs, signed image links, asynchronous jobs with signed webhooks and bulk capture of up to 100 URLs per call. Its response identifies page verdict and billing: bot checks or CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed. An MCP server provides take_screenshot, get_page_info and capture_pdf for Claude, Cursor and other MCP clients. The Free plan includes 1,000 screenshots per month without a card; paid plans start at $5 for 3,000.
Sign up free for ScreenshotNeo and start with the 1,000 monthly screenshots included without a card.
Best Value
FAQ
Is Scrapy a browser automation tool?
No. Scrapy is a crawler framework. Add scrapy-playwright when a real browser is needed.
Does Crawlee require Apify hosting?
No. Crawlee for Python can run anywhere; Apify deployment is an available option.
Which framework is fastest?
No neutral 2026 evidence establishes a universal winner. Test the workload you actually need to crawl.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Should every scraper use a proxy?
No. Proxy or session management is workload- and permission-dependent. Add it only when lawful, necessary and compatible with the target’s rules.
The Bottom Line
Choose Scrapy for the default multi-page HTML crawl, Crawlee for a unified Python HTTP/browser workflow, and browser automation when JavaScript interaction is unavoidable. Keep browser use targeted, build in queueing and politeness, and evaluate total infrastructure cost rather than the framework’s install price alone.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
