To screenshot an infinite-scroll page with Scrapy, use scrapy-playwright to scroll the page until the target feed has finished loading, then call Playwright’s screenshot(full_page=True). The screenshot option captures the page’s current full scrollable extent; it does not load items that have not appeared yet. First check whether the page’s data can be fetched directly—Scrapy recommends reproducing the data request when possible, while a browser is useful when you need rendered content or a screenshot. Scrapy’s dynamic-content guide and the Playwright Page API document these approaches.
Choose between fetching the data and rendering the page
Before adding a browser to a crawler, inspect the page’s network activity and markup. Infinite feeds often request additional data as the visitor scrolls. If you can identify and reproduce the request that returns the desired items, Scrapy can fetch and parse that data without rendering every browser-visible state. Scrapy’s documentation says reproducing requests containing the desired data is the preferred approach for pages that fetch data from additional requests.
As an Amazon Associate I earn from qualifying purchases.
| Situation | Approach | Trade-off |
|---|---|---|
| The feed data is available from an API, embedded JSON, or a reproducible request. | Use Scrapy’s ordinary request and parsing workflow. | Inspect the actual request and reproduce its relevant method, URL, body, and headers. This avoids browser rendering, but it does not produce a browser screenshot. |
| You need a screenshot or the useful content only appears in the rendered DOM. | Use a headless browser. | It can capture browser-visible state, but adds browser setup and lifecycle management. |
| You need browser actions as part of a Scrapy crawl. | Use scrapy-playwright. |
It integrates browser actions with Scrapy’s request processing. Scrapy recommends it for better integration. |
| You need a one-off browser script and do not need Scrapy’s crawl components. | Direct Playwright may be suitable. | Using Playwright directly inside a spider can bypass Scrapy components such as middleware and duplicate filtering. |
The official documentation does not establish a fixed speed or reliability advantage for one route. Choose based on whether you need the browser-rendered page or only its underlying data.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchInstall and configure scrapy-playwright
The scrapy-playwright README lists Python 3.10, Scrapy 2.7, and Playwright 1.40 as minimum versions. These requirements can change, so check the project README against your installed versions before setup.
#1 Best Overall
- Install the integration:
pip install scrapy-playwright. - Install browser binaries:
playwright install. - In your Scrapy settings, register the download handler and asyncio reactor. Chromium is the integration’s documented default browser type.
DOWNLOAD_HANDLERS = {
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"Registering the HTTPS handler is usually enough for modern sites, according to the project README. The asyncio-based reactor is the default in new projects since Scrapy 2.7; check existing project settings and your installed Scrapy version before changing the reactor.
Scroll until the feed is loaded, then capture it
The page-specific scroll loop is the critical part. Scroll in bounded increments, wait for a relevant response or DOM change, and stop when you observe an end-of-feed marker or a conservative no-growth condition. Keep a hard iteration or time limit. There is no universal scroll count or wait duration that works for every site.
This spider shows the integration pattern. The scroll_until_feed_is_loaded function is intentionally left to implement for the target site: its end condition depends on that page’s behavior. The example uses Scrapy’s asynchronous start method and an async callback, includes the Playwright Page for interaction, and closes it even if scrolling or capture fails.
import scrapy
class ScreenshotSpider(scrapy.Spider):
name = "screenshots"
async def start(self):
yield scrapy.Request(
"https://example.org/long-feed",
callback=self.capture,
meta={"playwright": True, "playwright_include_page": True},
)
async def capture(self, response):
page = response.meta["playwright_page"]
try:
await scroll_until_feed_is_loaded(page)
await page.screenshot(path="feed.png", full_page=True)
finally:
await page.close()Replace the example URL and implement the stopping logic using the target page’s actual behavior. For example, wait for a known final marker or item count if the page exposes one. Otherwise, compare the relevant item count or content between bounded scroll attempts and stop after several checks show no new items. A fixed sleep alone is fragile because response and rendering times vary. Page height alone may also be misleading when a site updates or recycles content without changing document.body.scrollHeight.
Why loading and capturing are separate
Playwright’s full_page=True option captures the full scrollable page as it exists at capture time rather than only the visible viewport. It does not make the site fetch every item in an infinite feed. Run and verify the scrolling logic first; then take the screenshot. Likewise, full-page capture or scrolling alone does not guarantee that every lazy-loaded image has finished loading.
Page lifecycle in Scrapy
Set meta={"playwright": True} to use the integration. When the callback needs to interact with the page, also set playwright_include_page=True; in an async callback the Page is available as response.meta["playwright_page"]. The integration closes a page after processing when it was not included. For an included page, close it yourself in a finally block so failures do not leave browser resources open. See the project README for the integration’s request and page behavior.
Common failures and how to fix them
- The capture contains only the initial viewport or current page extent. Confirm the scrolling function ran before capture and that additional feed items appeared in the DOM.
full_page=Truecaptures what is loaded; it does not trigger more loading. - The feed loads only once, or not at all. Wait for the page’s actual response or content change rather than assuming a delay. Check whether the page uses a “Load more” button or a nested scroll container instead of the window.
- The scroll loop never ends. Use an explicit end marker where possible, several unchanged-item checks otherwise, and a hard iteration or time cap. Do not rely on document height as the only signal.
- Cards appear but images are missing. Check that scrolling brought lazy-loaded assets into view and allow them to load before capture. The documented scroll and screenshot APIs do not promise to eagerly load every image.
- Scrapy middleware or duplicate filtering seems bypassed. Use
scrapy-playwrightwithin the spider instead of creating an independent Playwright browser when Scrapy’s request scheduling and components matter. Scrapy’s dynamic-content guide discusses this integration choice. - Browser resources accumulate. Close included Page objects in
finally. For memory-intensive pages, consider reducing browser concurrency; the appropriate limit depends on the crawler and target pages.
Performance, reliability, and cost considerations
Directly reproducing a data request generally avoids the work of rendering a browser page, while a browser is necessary for a browser screenshot or some rendered-only content. The official sources cited here do not publish a performance comparison, so there is no supported universal speed multiplier or reliability ranking. Browser workload and resource use depend on the page and crawler configuration.
Make the scroll termination condition observable and bounded: use a known end marker or item count where available, track content changes, and include a maximum. This limits wasted work if the site stops loading or changes its behavior. For screenshots, also verify that the final item and needed assets are present before capture; a successful screenshot call does not establish that the entire conceptual feed has been loaded.
Best Value
Or skip the browser setup
If your goal is a screenshot rather than a Scrapy crawl, ScreenshotNeo provides a website screenshot API and MCP server. A GET request can return a PNG, JPEG, WebP, or PDF. Its clean-shot steps accept cookie or consent banners like a visitor and remove more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and responses identify the page verdict and billing status in headers. AI agents can use its MCP server tools, including take_screenshot, get_page_info, and capture_pdf.
Example cURL request:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.org/long-feed -o shot.webpSee the ScreenshotNeo API documentation for request options and response details. One thousand screenshots per month are free with no card; paid plans start at $5 for 3,000. Sign up for the free plan.
Frequently Asked Questions
Does Playwright’s full-page screenshot load every item in an infinite feed?
No. It captures the full scrollable page as it exists when called; the page must be scrolled and its content loaded first.
Recommended Free Tools
Can I use direct Playwright in a Scrapy spider?
You can, but Scrapy warns that direct use can bypass Scrapy components such as middleware and duplicate filtering; it recommends scrapy-playwright for better integration.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
