The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →To scrape a website, request a page, inspect the response, and extract only the fields you need. Start with ordinary HTML; if the data is missing, look for a permitted data source the page uses, and use browser rendering only when the content is available only after JavaScript runs. Scraping one page is different from crawling: crawling follows links to more pages.
Before you scrape: define the target and check permission
Choose pages and fields
Write down the pages you need and the specific fields to collect—for example, a page title, a product name, or a publication date. Start with one representative page. Keeping the scope narrow makes it easier to select the right extraction method and check whether the results are accurate.
Read the site’s instructions and terms
Check the site’s current terms and look for its robots.txt, usually at the domain root, such as https://example.com/robots.txt. Robots rules tell crawlers which paths the site asks them to avoid. They are not permission to access content: RFC 9309 says, “These rules are not a form of access authorization.” Read the IETF’s Robots Exclusion Protocol (RFC 9309) and MDN’s explanations of robots.txt and X-Robots-Tag. The latter directives concern indexing and search presentation, not crawler authorization.
Use public pages and authorized interfaces. Check applicable law and privacy obligations before collecting, storing, or reusing content. Whether a specific scrape is lawful or allowed depends on the site, the data, and the circumstances; robots.txt alone cannot answer that.
Recommended Free Tools
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Choose the simplest method that exposes the data
| What you find | Approach | Trade-off |
|---|---|---|
| The required fields appear in the HTTP response’s HTML. | Make a direct HTTP request and parse the HTML. | Lightweight, but selectors can break when the page structure changes. |
| The site has many related pages, links, or pagination to process. | Use a crawling framework such as Scrapy. | Provides request-and-response handling for a crawl, but adds setup and maintenance. |
| The response lacks the fields, but the page requests data from another source. | Inspect the page’s network activity and, if permitted, request the relevant data source. | May avoid rendering a browser; the endpoint or response format can change. |
| The content appears only after browser-side JavaScript runs. | Render the page with a headless browser. | Can expose the rendered DOM, with extra browser runtime and operational complexity. |
Scrapy recommends finding the data source when desired content is dynamically loaded, and using a headless browser as a fallback when the content remains available in the browser DOM. That is a decision path, not a guarantee that one method is always faster or more reliable. See Scrapy’s guide to selecting dynamically loaded content and MDN’s overview of JavaScript network requests.
Extract fields from ordinary HTML
Inspect the response before writing selectors
A browser can show content that a simple HTTP request does not receive. First request one page and inspect the returned HTML. If the expected text or attributes are present, parse that response rather than starting with a browser automation setup.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Here is a small Python example using requests and Beautiful Soup. Install the dependencies with python -m pip install requests beautifulsoup4, save the script as scrape.py, and run python scrape.py. Replace the example URL and selectors with ones for a page you are authorized to access.
import json
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
url = "https://example.com/article"
response = requests.get(
url,
headers={"User-Agent": "ExampleResearchBot/1.0 (contact: [email protected])"},
timeout=20,
)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
title = soup.select_one("h1")
record = {
"source_url": response.url,
"title": title.get_text(" ", strip=True) if title else None,
"canonical_url": (
urljoin(response.url, soup.select_one('link[rel="canonical"]')["href"])
if soup.select_one('link[rel="canonical"]')
and soup.select_one('link[rel="canonical"]').get("href")
else None
),
}
print(json.dumps(record, ensure_ascii=False, indent=2))This extracts an h1 and a canonical link when present, while retaining the final response URL for verification. A missing field becomes null instead of causing the script to fail. Change the selectors to match the page’s actual structure; there is no universal selector or output format for scraping.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Keep extraction narrow and results verifiable
- Select stable elements that identify the fields you actually need.
- Extract text or attributes deliberately, and normalize whitespace where appropriate.
- Keep the source URL alongside every record so you can check it against the page.
- Expect optional fields and unusual formatting; handle missing values rather than assuming every page is identical.
When content is dynamic: follow the data before rendering a browser
Check whether the page requests data
If a field is absent from the response, inspect the page’s network activity while loading it. A JavaScript-driven page may request JSON or another data format and update part of the page without navigating to a new document. If the relevant source is accessible and you are allowed to use it, requesting that data directly can be simpler than reproducing the whole browser experience.
Use browser rendering as a fallback
If the data source is unavailable or unsuitable but the content appears in the browser DOM, render the page in a headless browser and extract from the rendered DOM. This adds a browser runtime and more operational complexity. Do not assume a screenshot or a successful page load proves that the specific fields you need were extracted; validate the values themselves.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Screenshot option for visual capture
If the deliverable is a visual record rather than structured text, ScreenshotNeo is a website screenshot API and MCP server from Yorker Media. Its screenshot capture can return PNG, JPEG, WebP, or PDF; it is not a replacement for extracting structured fields from page content.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Crawl only the pages you need
When the task extends beyond one page, follow only relevant links and pagination. Keep request volume proportionate, and stop or reduce activity when the site signals a problem. There is no universal safe request rate established here; choose a cautious rate for the target and its terms rather than treating an arbitrary number as a standard.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Page structures can change, pagination can repeat or end unexpectedly, and some records may omit fields. Validate a sample of extracted records against their source pages, including pages with missing fields or unusual formatting, before relying on the output.
Troubleshoot common scraping failures
- The expected field is missing: Inspect the raw HTTP response. If the content is absent there, check whether the page loads a data source; use browser rendering only if needed and permitted.
- A selector returns no result: Confirm the selector against the current page HTML or rendered DOM. The site may have changed its markup, or you may be parsing the response rather than the rendered page.
- The request fails or times out: Check the URL, network connectivity, response status, and timeout. Avoid rapid retries; reduce request activity if the site is returning errors or otherwise signaling a problem.
- Some records have blank values: Treat fields as optional, handle missing elements explicitly, and compare affected records with their source pages.
- Pagination misses or repeats pages: Verify the next-page link or pagination data on representative pages, record visited URLs, and stop when there is no valid next page.
- The site restricts access: Do not treat a reachable URL or a robots.txt rule as authorization. Check the site’s terms and use an authorized API or dataset where available.
Or skip the browser setup
If you need a visual capture, ScreenshotNeo takes a screenshot with one GET request. For structured text extraction, use the HTML or data-source workflow above instead.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com -o shot.webpSee the ScreenshotNeo documentation for request options. Before capture, it accepts the cookie or consent banner like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and each response reports the page verdict and billing status in headers. Its MCP server provides take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 screenshots.
Sign up for ScreenshotNeo’s free plan.
Frequently Asked Questions
Is scraping the same as crawling?
No. Scraping extracts selected information from pages; crawling follows links to additional pages.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesDoes robots.txt give permission to scrape a page?
No. It gives crawler instructions, not access authorization. Check the site’s terms and applicable requirements separately.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
