Do not automate data extraction from Yellow Pages unless Thryv has given you prior express consent. Yellow Pages’ Terms of Use prohibit bots, scrapers, crawlers and similar tools from gathering or extracting data from its sites without that consent. If you need business names, contact details or categories, first get permission or use a data source whose license permits your intended collection and use. This guide explains how to check the rules and build a basic extractor for an authorized source—not how to bypass Yellow Pages’ restrictions.
What “scrape Yellow Pages” means—and the access rule
People searching for how to scrape Yellow Pages usually want to turn directory listings into structured records: business names, phone numbers, addresses, websites or categories. The technical steps—requesting a page, parsing its HTML and saving fields—are straightforward. The important first question is whether you are allowed to automate that collection from the particular site and for your intended purpose.
YellowPages.com describes its YP Sites as consumer business search and comparison services. Its Terms of Use grant a limited right to use them for individual, non-commercial informational purposes, subject to the terms and instructions that apply. They separately prohibit automated extraction without Thryv’s prior express consent. The terms state: “You may not use bots, scrapers, crawlers, spiders, or any similar methods, processes, or tools to ‘data mine’ or otherwise gather or extract data from the YP Sites, and you may not frame or proxy the YP Sites or utilize any other techniques to re-display the YP Sites (or any content on the YP Sites) without Thryv, Inc.’s prior express consent, which consent, if given, may be withdrawn by us at any time, with or without notice, in our sole discretion.” Read the YellowPages.com / Thryv Terms of Use before planning automated access; the terms and any service-specific instructions are authoritative for your use.
- A listing being visible in a browser does not, by itself, authorize automated collection.
- Manual access or a technically accessible page does not override the terms.
- Do not treat a robots.txt file as permission to extract data.
- The terms say Thryv may terminate access for a breach and may use technical barriers to prevent unauthorized access.
This is an access-and-permission constraint, not a recommendation to find a different way around a block. Do not use proxies, browser automation, altered request patterns or other techniques to defeat access controls or continue collection after a restriction.
#1 Best Overall
Get permission or choose a licensed source first
Ask Thryv about your specific use
The terms refer to API terms “where available,” but that reference does not establish that a generally available Yellow Pages API or bulk-data license exists for every user or use case. Contact Thryv directly to ask whether it can authorize your proposed project. Explain your geography and purpose, and ask for written terms covering the fields you need, the pages or services in scope, collection volume, retention, redistribution and any technical conditions. Do not assume that an API or license is available until Thryv confirms it for your situation.
If consent is granted, keep a copy and translate its conditions into a collection plan before writing code. Record the approved source pages, fields, request volume, timing, retention period and downstream uses. Limit your extractor to that scope; if your intended use changes, check whether the authorization still covers it.
Evaluate another business-data provider
If you cannot obtain permission for Yellow Pages, look for a provider or dataset whose license expressly allows your planned access and use. Check the actual license rather than relying on a marketing statement that data is “public” or “available.” Compare providers on:
- Permission for automated access and the exact uses allowed.
- Countries, regions, categories and business types covered.
- Available fields, including whether contact information is included.
- Update cadence and how outdated or corrected records are handled.
- Retention, redistribution, attribution and deletion conditions.
- Price and any limits on volume, seats or commercial use.
These details vary by provider and license; verify them directly before building around a source. This article does not establish a particular alternative provider as suitable for every project.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What robots.txt can—and cannot—tell you
Robots.txt communicates crawler instructions. It is not a permission grant, and it does not replace a site’s terms or any required consent. A permissive robots.txt therefore does not authorize scraping Yellow Pages under the Terms of Use. Google’s documentation explains how Google crawlers fetch and parse robots.txt; it describes crawler handling, not a general license for other parties to collect site data. See Google’s explanation of the robots.txt specification.
For an authorized source, check its applicable terms and crawler instructions before collection, and follow any conditions in your permission. Robots.txt is one operational signal to review, not a substitute for contractual authorization.
Build a small extractor for a source you are allowed to process
The example below demonstrates parsing and exporting business records from an HTML file you are authorized to use. It does not contact Yellow Pages. Starting with a local file makes the parsing and data-cleaning steps easy to validate without requesting pages from a site. For an approved source, obtain its permission first, then adapt the file input and selectors to the HTML and conditions that source allows.
1. Install the parser
Use Python 3 and install Beautiful Soup:
python -m pip install beautifulsoup42. Save an authorized HTML sample
Create a file named listings.html containing a permitted sample. This small fixture uses ordinary HTML attributes and is included so the script can run as written:
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →<!doctype html>
<html>
<body>
<article class="business">
<h2 class="name">Northside Bikes</h2>
<a class="phone" href="tel:+15550101010">(555) 010-1010</a>
<a class="website" href="https://example.org/northside">Website</a>
<p class="category">Bicycle shop</p>
</article>
</body>
</html>3. Parse, validate and write CSV
Save this as extract_listings.py in the same directory and run python extract_listings.py. The script reads only the local file, normalizes whitespace, skips records without a name and deduplicates identical rows before writing businesses.csv.
import csv
from pathlib import Path
from bs4 import BeautifulSoup
source = Path("listings.html")
html = source.read_text(encoding="utf-8")
soup = BeautifulSoup(html, "html.parser")
def clean_text(element):
return " ".join(element.get_text(" ", strip=True).split()) if element else ""
records = []
seen = set()
for card in soup.select("article.business"):
name = clean_text(card.select_one(".name"))
if not name:
continue
phone_link = card.select_one("a.phone")
website_link = card.select_one("a.website")
record = {
"name": name,
"phone": clean_text(phone_link),
"phone_href": phone_link.get("href", "") if phone_link else "",
"website": website_link.get("href", "") if website_link else "",
"category": clean_text(card.select_one(".category")),
}
key = (record["name"].casefold(), record["phone_href"], record["website"])
if key not in seen:
seen.add(key)
records.append(record)
fields = ["name", "phone", "phone_href", "website", "category"]
with Path("businesses.csv").open("w", newline="", encoding="utf-8") as output:
writer = csv.DictWriter(output, fieldnames=fields)
writer.writeheader()
writer.writerows(records)
print(f"Wrote {len(records)} records to businesses.csv")For a licensed HTML source, inspect an authorized sample and update article.business, .name, a.phone, a.website and .category to match its documented structure. Do not infer that selectors shown here match Yellow Pages’ current pages. Keep only fields covered by your permission and the source’s terms. Before using records, validate field formats and decide how your application will handle incomplete or stale entries.
Fetching pages: only when the source authorizes it
If the source’s terms and your permission allow automated HTTP requests, the fetch stage can use a standard HTTP client. The following pattern is for an authorized endpoint only; it is not a Yellow Pages recipe. Replace the URL and selectors only with those for a source you may process. Follow the source’s documented rate limits and permission conditions rather than trying to evade a restriction.
import requests
from bs4 import BeautifulSoup
url = "https://authorized.example/path"
response = requests.get(
url,
headers={"User-Agent": "BusinessDataProject/1.0 contact: [email protected]"},
timeout=(5, 20),
)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
for card in soup.select("article.business"):
name = card.select_one(".name")
if name:
print(name.get_text(" ", strip=True))The example uses an intentionally non-operational domain because the right authorized endpoint depends on your provider and permission. For a real deployment, use the provider’s published access method when available, set timeouts, handle HTTP errors, and store only permitted data. If the source returns an error or blocks access, stop and resolve the issue with its operator; do not rotate identities or disguise traffic to get around it.
Recommended Free Tools
Best Value
Data quality, performance and responsible operation
Make records useful without collecting more than you need
Choose a stable key for deduplication based on the fields you are authorized to retain. A name alone is often not unique; a name combined with a phone number or provider identifier can be more useful when available. Preserve the source or collection date if your permission allows it, so your application can distinguish recent records from old ones. Validate phone numbers, URLs and required fields before downstream use; malformed HTML and missing values are normal cases for parsers to handle.
Keep the collection rate within the authorization
For a permitted source, use the lowest request rate that meets the project’s need and any limits set by the provider. If the provider documents a request limit, honor it. Cache or reuse permitted results where the license allows, rather than repeatedly fetching the same pages. If a request times out or returns an error, record the failure and use bounded retries only where the provider’s conditions permit them. Avoid concurrent or repeated requests that exceed the approved scope.
Plan for changes and failure
Page markup can change, causing a selector to return empty values or capture the wrong text. Test a small authorized sample, check required fields, and stop the job if validation indicates that the page structure has shifted. Keep logs of request status and parsing outcomes, without logging credentials or collecting unapproved personal data. For larger workflows, separate fetching, parsing, validation and export so a parser problem does not silently produce a bad dataset.
Troubleshooting an authorized extraction workflow
- Python reports that Beautiful Soup is missing: install it in the same Python environment used to run the script with
python -m pip install beautifulsoup4. - The script finds zero records: confirm the input file path and inspect whether the sample contains the selectors in the script. For an authorized provider, its page structure may differ; update selectors from a permitted sample rather than guessing.
- CSV rows have blank fields: verify that those fields exist in the source markup and that the selector targets the correct element. Treat absent data as missing; do not fabricate values.
- The HTTP request times out or returns an error: check the endpoint, network connectivity and provider status. Respect any access restriction and contact the source operator if authorization or permitted access is unclear.
- The site returns a challenge or blocks the request: stop automated collection and ask the operator about an approved access route. Do not add proxies, browser automation or other bypass measures.
- The output contains duplicate or stale listings: revise the deduplication key and refresh policy only within the source’s license and your approved retention terms.
Or skip the browser setup
ScreenshotNeo is a website screenshot API and MCP server, not a Yellow Pages data extractor and not permission to collect directory records. If your task is to capture an authorized page visually, its API can return a screenshot or PDF; use the product only for pages you are allowed to access and capture. The one-request example below captures ScreenshotNeo’s own public page. See the ScreenshotNeo documentation for API options.
Quick Recap
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://screenshotneo.com -o shot.webpScreenshotNeo removes cookie and consent banners, newsletter popups and chat widgets before the shot; each step can be turned off. Bot checks, blank pages, timeouts, failed loads and cache hits are not billed, and response headers say which outcome occurred. Its MCP server offers take_screenshot, get_page_info and capture_pdf for AI agents and MCP clients. The Free plan includes 1,000 shots per month without a card; paid plans start at $5 for 3,000 shots. See ScreenshotNeo for product details, or sign up free for 1,000 screenshots a month with no card.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
