Beautiful Soup parses HTML; it does not fetch web pages or run JavaScript. A typical scraper uses an HTTP client such as Requests or Python’s urllib.request to retrieve permitted markup, then passes that markup to Beautiful Soup to locate and extract data. This tutorial builds that workflow, explains parser choices and common failures, and shows when static-page parsing is not enough.
What is web scraping?
Web scraping is the process of retrieving information from web pages and extracting selected data from their contents. A basic Python scraper has distinct stages: request a page, receive its response, parse the HTML into a structure, find the elements that contain the information, and save only the fields you need.
As an Amazon Associate I earn from qualifying purchases.
Beautiful Soup’s project documentation describes it as “a Python library for pulling data out of HTML and XML files.” It turns markup into a navigable tree; it is not the network client. Keeping those jobs separate makes it easier to diagnose whether a problem comes from the request or the parsing.
What is the difference between requests and BeautifulSoup?
Requests (the Python package is conventionally imported as requests) sends an HTTP request and gives your program a response. Beautiful Soup, imported from bs4, parses the response body. The response might contain a page, an error, a redirect result, or markup that differs from what you expected; Beautiful Soup cannot correct a failed request or retrieve missing content by itself.
#1 Best Overall
Martin Breuss’s Real Python tutorial, published December 1, 2024, describes Requests as a user-friendly way to scrape static HTML and Beautiful Soup as the package used to parse that HTML. The division remains a useful mental model: retrieval first, parsing second.
Check permission and choose a suitable target
Before making requests, read the target site’s terms and its robots.txt, and confirm that the pages and paths you plan to access are permitted. These checks are practical safeguards, not a complete legal determination; rules can depend on the content, your use, and the relevant jurisdiction. If the site disallows the access, stop rather than trying to work around its controls.
- Learn with a practice site intended for scraping or a local HTML file.
- Request only the pages you need, at a considerate rate, and extract only the fields needed for your task.
- Avoid personal data and content behind a login unless you have a clear, authorized basis to access and use it.
- Do not use altered headers or other tactics to disguise your client or bypass access restrictions.
Install Beautiful Soup and a parser
For new projects, install the beautifulsoup4 distribution and import the class from bs4. Do not install a similarly named BeautifulSoup package by mistake: the project manual identifies that as the old Beautiful Soup 3 line, which is no longer developed or supported.
python -m pip install beautifulsoup4 requests lxml
This installs Requests for fetching pages and lxml as an optional parser dependency. If you prefer Python’s built-in parser, omit lxml and specify html.parser in your code. Consult the Beautiful Soup documentation for the current manual and check your installed package version when version-specific behavior matters. The manual retrieved October 7, 2026, is labeled Beautiful Soup 4.14.3; that page label is not a release-date claim.
Fetch a page and parse its HTML
This example uses Requests and a placeholder for a practice page you are authorized to access. Replace the URL with a permitted target. It checks for an unsuccessful HTTP response before parsing and explicitly selects a parser so the resulting tree is more consistent across environments.
import requests
from bs4 import BeautifulSoup
url = "https://example.com/permitted-practice-page"
response = requests.get(url, timeout=20)
response.raise_for_status()
soup = BeautifulSoup(response.text, "lxml")
print(soup.title.get_text(strip=True) if soup.title else "No title found")
A timeout prevents the request from waiting indefinitely; raise_for_status() raises an exception for unsuccessful HTTP status codes rather than letting the script silently parse an error page as if it were the intended content. For a known encoding or unusual response, inspect the response headers and the page’s actual markup before deciding how to decode it.
Python’s standard library can also make a request. Its urllib.request.Request object can carry headers and an HTTP method; when no request data is supplied, GET is the default. Use headers for ordinary, truthful request configuration—not to evade a site’s restrictions.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
request = Request("https://example.com/permitted-practice-page", method="GET")
with urlopen(request, timeout=20) as response:
html = response.read()
soup = BeautifulSoup(html, "html.parser")
See the Python 3.13.16 urllib.request documentation for request configuration details. For broader usage, including response handling, the Beautiful Soup manual documents parsing and navigation.
Choose a parser deliberately
Beautiful Soup supports lxml, html5lib, and Python’s built-in html.parser. They can produce different trees from malformed HTML because they repair and interpret that markup differently. There is no universally correct result for invalid input. The project manual recommends naming the parser explicitly, particularly when code will run on multiple machines or be distributed.
| Parser | What to expect | Practical consideration |
|---|---|---|
lxml |
The manual ranks it first among these parser choices and describes it as significantly faster than the other named parsers. | Install the lxml dependency and keep it available in each environment where the code runs. |
html5lib |
Uses HTML5 parsing techniques; malformed input may produce a tree different from other parsers. | Install html5lib and select it explicitly when its HTML5-oriented interpretation suits the input. |
html.parser |
Python’s built-in HTML parser; it can yield a different tree from lxml or html5lib on malformed markup. | No separate parser package is required, but specify it rather than relying on automatic selection. |
Do not switch parsers casually after writing selectors: a changed tree can alter which elements match. For repeatable extraction, choose a parser, declare its dependency where relevant, and verify your selectors against the pages you actually process. The manual’s speed comparison is qualitative; it does not provide a benchmark figure for these options.
Find elements and extract text or attributes
Once parsed, locate elements by tag and attributes, or use CSS selectors. Extract visible text with get_text(); read an attribute such as a link destination from the element’s attribute mapping. Always check whether a match exists before accessing it, because page structure can change or the expected element may be absent.
Free tools Windows power users keep installed
One-click scans. No signup required.
article = soup.select_one("article h1")
if article is None:
raise ValueError("Expected article heading was not found")
heading = article.get_text(" ", strip=True)
link = soup.select_one("article a")
if link is not None:
href = link.get("href")
link_text = link.get_text(" ", strip=True)
print(heading, href, link_text)
select_one() returns the first match or None; select() returns all matching elements. For repeated records, iterate over the matches and validate fields within each record rather than assuming every item has identical markup.
Best Value
records = []
for card in soup.select(".result-card"):
title = card.select_one(".result-title")
if title is None:
continue
records.append({
"title": title.get_text(" ", strip=True),
"url": (card.select_one("a") or {}).get("href"),
})
For robust production code, be careful with fallback logic like the link expression above: a missing anchor should be handled explicitly if its absence is an error. Normalize text consistently, preserve the source URL when useful, and save only the fields your task requires. Validate output before relying on it downstream.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Why does my scraper return an empty list?
An empty result usually means that the markup you parsed does not contain the elements your selector expects. That can happen even when the request succeeded. Diagnose the response and the parsed HTML before changing selectors or adding tools.
- Inspect the response. Check the status code, final URL, and response body. A successful HTTP response can still be a consent page, an error template, or a different page than expected.
- Look for the target in the raw HTML. Search the response text for a distinctive piece of the expected content. If it is absent, the problem is upstream of Beautiful Soup’s selector.
- Check the selector against the actual markup. Confirm tag names, class names, nesting, and attributes. A site redesign or slightly different page template can invalidate a previously correct selector.
- Confirm the parser. Try parsing with the explicitly chosen parser used in your project and inspect the relevant part of the resulting tree. Malformed HTML can be interpreted differently by different parsers.
- Handle missing matches. Use
select_one()or a similar lookup, test forNone, and log or report the missing field rather than silently producing incomplete records.
If the page’s content is not in the response HTML because it is added after page load, changing a Beautiful Soup selector will not make it appear.
What if the content is added by JavaScript?
Beautiful Soup parses markup; it does not execute JavaScript or render a browser DOM. If the desired text is missing from the fetched HTML, first check whether the site offers an official API or data export. That is often a more direct, structured source than scraping a rendered page.
If the content genuinely depends on rendered browser state and access is permitted, a suitable browser automation or rendering tool may be necessary. That is a separate step from parsing: you still need to inspect the resulting content, extract carefully, and honor the site’s access conditions. Real Python’s tutorial distinguishes static HTML from dynamic pages and discusses the need for additional tools in some dynamic-page cases.
Further learning
The official Beautiful Soup manual is the primary reference for installation, parser selection, navigation, and extraction. For a guided explanation of scraping static pages and the distinction between static and dynamic content, see Martin Breuss’s Real Python Beautiful Soup tutorial, published December 1, 2024. For request configuration using Python’s standard library, consult the urllib.request documentation for Python 3.13.16.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
