Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsTo capture network requests in browser automation, attach request and response listeners before navigating or performing the action that triggers traffic. Use Playwright events for ongoing observation, waitForResponse() for one anticipated API response, and routing when you need to modify, fulfill, or abort a request. In Selenium, use WebDriver BiDi when the browser and language binding support the network events you need; use Chrome DevTools Protocol (CDP) for lower-level, Chrome-specific access.
Choose what to capture
“Scraping network requests” can mean several different things. Choose the level of access before writing the listener:
As an Amazon Associate I earn from qualifying purchases.
- Request metadata: record a URL, HTTP method, or resource type to see what the page sends.
- Response metadata: associate a request with its response and record the HTTP status or headers.
- Response body: read the data returned by an endpoint. Wait until the response body has been received, and consult the framework’s API for its body-reading method.
- Network interception: pause a request so the test can modify it, provide a response, or abort it. This changes traffic rather than merely observing it.
For a continuous log, install listeners before navigation. For one expected endpoint, wait for the matching response before triggering the click or other action. Use interception only when the test needs to affect the request.
Capture network requests with Playwright
Playwright can emit request and response events for page activity, including XHR and fetch traffic. Register the handlers before goto() or the user action that causes the request:
#1 Best Overall
- Multifunctional Network Cable Tester: TESMEN TLP-123A Supports RJ45 and RJ11, enabling rapid detection of line connectivity, short circuits, open circuits, miswiring, and cable shielding status. An essential tool for troubleshooting line faults and network maintenance, it effectively boosts your work efficiency
- Convenient and Efficient: Featuring one-button operation and a test speed adjustment gear on the main control unit for enhanced flexibility. Clear LED indicators provide intuitive test result displays, making it easy for both professionals and home users to operate
- Portable and Durable: Compact and lightweight design for easy portability. Constructed with high-quality plastic housing for robust structure, ensuring both durability and stability. Ideal for home wiring, IT equipment setup, electrical maintenance, and LAN DIY projects
- Detachable design: The main control unit and remote unit can be separated and used independently, allowing you to test both ends of long cables. This makes it ideal for wall-mounted ports, long-distance cabling, or structured cabling systems, perfect for homes, offices, or professional IT environments
- What you will get: 1 * TLP-123A Network Cable Tester, 1 * user manual, 2 * AAA batteries
page.on('request', request => {
console.log('>>', request.method(), request.url());
});
page.on('response', response => {
console.log('<<', response.status(), response.url());
});
await page.goto('https://example.com');This logs request method and URL, then response status and URL. It does not log response bodies. The response event means the response status and headers have arrived; the request finishes after its body has downloaded.
Wait for one request or response
When a click should trigger a known API call, create the wait promise first, then perform the action and await the result:
const responsePromise = page.waitForResponse('**/api/fetch_data');
await page.getByText('Update').click();
const response = await responsePromise;
console.log(response.status(), response.url());The glob pattern is matched against the full URL. For complex matching, use a regular expression or predicate that checks the URL, method, or other relevant request details. A predicate example:
Rank #2
- VERSATILE CABLE TESTING: Cable tester for data (RJ45) terminated cables and patch cords, ensuring comprehensive testing capabilities
- LARGE BACKLIT LCD: Backlit LCD display enables easy reading of pin-to-pin wiremap results, even in low-lit areas
- COMPREHENSIVE FAULT DETECTION: Test for Open, Short, Miswire, Split-Pair faults, Cross-over, and Shield, providing thorough fault detection
- INTUITIVE USER INTERFACE: User-friendly interface with three buttons and simple, easy-to-identify test responses, ensuring a smooth testing experience
- MULTIPLE TONE GENERATOR STYLES: Tone on a single wire, wire pair, or all 8 conductor wires using the multiple style tone generator (solid/warble); requires probe Cat. No. VDV500-123 (sold separately)
const responsePromise = page.waitForResponse(response =>
response.url().includes('/api/fetch_data') &&
response.request().method() === 'GET'
);
await page.getByText('Update').click();
const response = await responsePromise;Use the response object’s Playwright API to read its body after the wait resolves. If you need to observe every request rather than one expected response, use event listeners instead.
Observe traffic or intercept it
Use page.on('request') and page.on('response') for passive observation. Use page.route() or browserContext.route() when you need to change a request, fulfill it with a test response, or abort it. A matching routed request stalls until the route handler handles it, so make sure every path through the handler continues, fulfills, or aborts the request as intended.
Understand request, response, and failure events
Think of the lifecycle as three distinct points:
- Request: the browser issues the request.
- Response: the browser receives response status and headers.
- Request finished: the response body has finished downloading.
If the browser cannot obtain an HTTP response—for example, because of a network error or timeout—the request may instead emit requestfailed. An HTTP error status such as 404 or 503 is still an HTTP response; it is not, by itself, a requestfailed event. Log the status to detect server-side errors, and treat transport failures separately.
Rank #3
- Versatile Cable Compatibility: Engineered for RJ45 Ethernet (CAT5/5e/6/6A/7) and RJ11 telephone lines, the HT812A provides reliable detection of continuity, short circuits, open lines, miswiring, and shielding conditions across multiple cable standards.
- Dual Scan Modes for Precise Diagnostics: Fast and Slow test modes enable flexible troubleshooting. LED indicators on both master and remote units clearly reflect pin-to-pin mapping, allowing accurate identification of wiring faults.
- Split-Module Architecture: The detachable master and remote units support long-distance and in-wall cable verification. Ideal for structured cabling, patch panels and distributed network installations.
- Robust & Compact Construction: Built with a durable ABS enclosure, the tester remains lightweight and easy to carry. Whether for technicians, DIY users, or electricians, this pocket-sized tool is perfect for quick diagnostics and routine maintenance.
- Safe Line Testing & Simple Control: One-switch operation ensures straightforward testing. LED output displays wiring status, while the shield indicator identifies FTP/STP vs. UTP cables. Supports safe testing on telephone lines within standard operating voltages.
Use Selenium WebDriver BiDi or Chrome DevTools Protocol
The right alternative depends on your existing automation framework, target browser, and whether you need event observation, body access, or interception. Support can vary by browser and language binding, so verify the precise operation against the versions you deploy.
| Approach | Best fit | Important qualification |
|---|---|---|
| Playwright events and routing | Playwright tests needing request/response events or high-level routing. | Native routing can be affected by Service Workers; see the troubleshooting section. |
| Selenium WebDriver BiDi | Selenium automation that needs bidirectional event streaming, including network events. | BiDi APIs and browser support are evolving. Enable the webSocketUrl capability using the syntax for your language binding, then confirm the required event or operation is available. |
| Chrome DevTools Protocol (CDP) | Direct access to Chrome’s Network domain when protocol-level operations are needed. | CDP is Chrome-specific, not a cross-browser standard. Selenium describes its CDP support as temporary while BiDi is implemented. |
Selenium BiDi setup considerations
WebDriver BiDi streams events bidirectionally over WebSockets. Selenium describes it as the W3C bidirectional direction for WebDriver, with capabilities growing as browser vendors implement the standard. Set the webSocketUrl capability in the form required by the language binding, then consult that binding’s current API for subscribing to the specific network events you need. Do not assume that one event API is available identically across all Selenium versions and browsers.
When CDP is the better fit
CDP exposes Chrome’s Network domain, including request and response events and methods for retrieving request post data or response bodies. Choose it when you need those lower-level Chrome protocol operations and can accept the browser-specific dependency. For browser-neutral automation, prefer the framework’s supported abstraction or BiDi where the deployed browser and binding expose the needed features.
Rank #4
- 【Five Gigabit Ports】1 Gigabit WAN Port plus 2 Gigabit WAN/LAN Ports plus 2 Gigabit LAN Port. Up to 3 WAN ports optimize bandwidth usage through one device.
- 【One USB WAN Port】Mobile broadband via 4G/3G modem is supported for WAN backup by connecting to the USB port. For complete list of compatible 4G/3G modems, please visit TP-Link website.
- 【Abundant Security Features】Advanced firewall policies, DoS defense, IP/MAC/URL filtering, speed test and more security functions protect your network and data.
- 【Highly Secure VPN】Supports up to 20× LAN-to-LAN IPsec, 16× OpenVPN, 16× L2TP, and 16× PPTP VPN connections.
- Security - SPI Firewall, VPN Pass through, FTP/H.323/PPTP/SIP/IPsec ALG, DoS Defence, Ping of Death and Local Management. Standards and Protocols IEEE 802.3, 802.3u, 802.3ab, IEEE 802.3x, IEEE 802.1q
Troubleshoot missing or misleading traffic
- The request never appears: attach listeners before navigation or the click that triggers it. If using
waitForResponse(), create the promise before the action. - The route handler never sees a request: check whether a Service Worker handles it. Playwright warns that native
page.route()andbrowserContext.route()may appear to miss traffic when Service Workers are active. If the test permits it, configure the context withserviceWorkers: 'block'. If you need to observe Service Worker requests themselves, use Playwright’s dedicated Service Worker guidance rather than blocking them. - The traffic belongs to another page: determine whether the action opened a popup or a separate page, and attach observation to the page or context that owns the request.
- The status is 404 or 503: treat it as an HTTP response and inspect its status. Do not classify it as a transport failure just because it is an error status.
- The wait times out: confirm that the action really triggers the request, the matcher covers the full URL, and the method or other predicate conditions are correct. Also check whether the response is issued by a popup or handled by a Service Worker.
- The response body is unavailable when the response event fires: the response event precedes completion of body download. Wait for request completion or the relevant response-reading API before processing the body.
- A routed request hangs: ensure the route handler resolves every matching request by continuing it, fulfilling it, or aborting it. Routed requests stall until handled.
Performance, reliability, and cost considerations
Continuous listeners are useful for diagnosing unexpected traffic, but they can produce large logs on pages with many requests. Filter by URL, method, or resource type when you only need a subset. For a known endpoint, a targeted wait avoids collecting unrelated events. Reading bodies can add work and retain sensitive or large payloads; capture only the data your authorized test needs, and avoid logging credentials, session cookies, or personal data.
Network observations depend on the browser, automation framework, protocol support, and page behavior. A test that uses BiDi or CDP should be checked against the exact browser and binding versions in the deployment environment. For reproducible results, record the request URL, method, response status, and relevant failure information, while keeping any body capture narrowly scoped.
Recommended Free Tools
Or skip the browser setup
ScreenshotNeo is a website screenshot API, not a network-traffic capture tool: it returns a screenshot or PDF, not request logs, response bodies, or interception events. If a visual capture is enough, one GET request can return a clean screenshot. Before capture, it accepts cookie or consent banners and removes 60+ known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing status. It also provides an MCP server for AI agents such as Claude, Cursor, and other MCP clients.
Example using cURL (see the ScreenshotNeo documentation):
Best Value
- Versatile Connectivity Testing: Expertly designed to test generic network and telephone cabling. Fully compatible with RJ45 and RJ11 interfaces, supporting a wide range of Ethernet standards including CAT5, CAT5e, CAT6, CAT6a, CAT7, and UTP/STP shielded cables. An essential diagnostic tool for home repair and IT network maintenance.
- Pinpoint Accuracy & Troubleshooting: Instantly verify your wiring integrity. The device runs automatic scans to detect opens, shorts, cross wires, and miswires. The clear LED array (1-8 and G) provides immediate visual feedback, allowing you to troubleshoot faulty connections or confirm a successful crimp in seconds.
- Smart Shielding Detection & Speed Control: Unlike basic testers, this unit features a specific "G" light to confirm ground shielding status for STP cables. It also includes an "S" Mode (Slow Mode), which slows down the LED sequence, making it easier to observe the lights individually and prevent missing any skipped pins.
- Detachable Design for Long Runs: The master and remote units slide apart easily, allowing you to test cables that are already installed in walls or ceilings across different rooms. Compact and encased in shock-resistant ABS, it fits perfectly in any pocket or tool bag.
- Battery Operated & Plug-and-Play: Powered by a standard 9V battery (not included) with a smart power-saving design for extended use. Operation is simple: just plug in your cable ends, switch to "ON" or "S", and get results instantly. No complex setup or software required.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com -o shot.webpFree includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots. Sign up for ScreenshotNeo’s free plan.
Frequently asked questions
Can I capture network requests without changing them?
Yes. Use Playwright’s request and response events, or the corresponding event capabilities exposed by your Selenium BiDi binding. Use routing only when you intend to intervene in traffic.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Does ScreenshotNeo capture API requests made by a page?
No. ScreenshotNeo captures a visual screenshot or PDF; it does not provide network request logs or response bodies.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
