Best User-Agent List for Scraping: Current Examples and Safe Rotation

The best User-Agent list for scraping is a small, maintained set of truthful profiles—not a huge pool of copied strings. Choose a User-Agent that matches the client behavior you actually send, keep it stable for a session, and rotate only at deliberate boundaries such as a new session or crawl partition. A changed header does not grant permission to access a site or guarantee fewer blocks.

What a User-Agent does—and what it does not do

A User-Agent (UA) is an HTTP request header that identifies a client to a server. Browsers use it to describe themselves; crawlers can use a clear product identity so site owners can recognize and contact the operator. The server may use the value when selecting content or applying rules, but the header is only one part of a request. Cookies, request rate, IP reputation, TLS behavior, JavaScript execution, and the site’s own access controls can also matter.

#	Preview	Product	Price
1		The Proxy Playbook: The Complete Guide to Proxy Servers: How to Source, Test, and Scale Residential,...	$29.95	Buy on Amazon

Changing a UA string changes that header. It does not turn a Python HTTP client into Chrome or Safari, remove other signs of the client, satisfy a site’s terms, or make a prohibited request acceptable. No universal block-rate improvement or ideal pool size is established for UA rotation. Treat rotation as a way to test or represent compatible clients—not as a bypass strategy.

A practical User-Agent list to start from

MDN’s reference examples show reduced browser identities: platform and browser details are intentionally less specific than many older lists suggest. The Chrome version below is a version-sensitive example, not a permanent current-version guarantee. Refresh browser versions from supported clients before deploying a profile.

#1 Best Overall

The Proxy Playbook: The Complete Guide to Proxy Servers: How to Source, Test, and Scale Residential, Mobile, and Datacenter Proxies for Web Scraping, ... Anonymous Browsing without Getting Blocked

$29.95

Buy on Amazon

The examples and patterns above follow MDN’s current reference material as represented here; browser versions are volatile. Chrome’s User-Agent reduction is complete, according to its official documentation: exact OS versions, device models, and minor browser versions are reduced. Additional details can be requested through User-Agent Client Hints when a server has a legitimate need and the client supports the opt-in flow. Old lists full of exact Android models or detailed minor versions may therefore be historical rather than useful identities.

How to choose and rotate profiles without creating contradictions

1. Choose identity before volume. If you operate a crawler, prefer a stable, descriptive crawler User-Agent with a product token, an information page, and contact details. If you need to test browser-specific behavior, select a profile that corresponds to the browser actually driving the request.
2. Keep the request coherent. A profile is more than the UA header: consider compatible Accept, Accept-Language, Accept-Encoding, cookie/session policy, and desktop or mobile behavior. Do not claim Safari while sending obviously different Chromium behavior. A header alone does not reproduce a browser.
3. Keep one profile sticky for a session. Select a profile at the beginning of a logical session or crawl partition and retain it with that session’s cookies and state. Do not switch identity on every request. If a change is needed, make it at an intentional boundary such as a new session, host, or rate window, and log the selection.
4. Preserve crawler identity across variants. RFC 9309 says a crawler’s product token should be a substring of the identification string it sends. Robots.txt matching is case-insensitive; matching groups are merged, and the wildcard group applies when no specific group matches. Keep the same product token in every custom crawler variant so the relevant robots rules remain applicable.
5. Refresh and retire entries. Maintain a short allowlist, record when and why a value was refreshed, and remove stale profiles. Validate actual behavior rather than assuming a plausible-looking string makes the client equivalent to a browser.
6. Treat access policy separately. Check the target’s robots.txt and terms, honor the applicable rules, rate-limit requests, and cache where allowed. A UA change is not permission to disregard access controls.

A crawler identity example

For a crawler you own, a practical format is ExampleResearchBot/1.0 (+https://example.org/bot-info). The token should remain stable; the linked information page should explain the crawler’s purpose and how to contact its operator. Scrapy’s documented default is Scrapy/VERSION (+https://scrapy.org), and its documentation recommends adding a project URL or email address so site operators have a contact route. The example domain above is illustrative—replace it with your own real information page, not a URL you do not control.

Python: check robots.txt and keep a profile sticky

This example is for a site you own or are permitted to crawl. It uses a stable crawler identity rather than pretending that the requests library is a browser. It checks the applicable robots rule before fetching a page and uses one session identity throughout the run. Replace the example host and crawler-info URL with real values; handle any applicable rate limits and site terms as well.

from urllib.parse import urlparse
from urllib.robotparser import RobotFileParser
import requests

BOT_UA = "ExampleResearchBot/1.0 (+https://example.org/bot-info)"


def robots_allows(url: str) -> bool:
    parts = urlparse(url)
    robots_url = f"{parts.scheme}://{parts.netloc}/robots.txt"
    parser = RobotFileParser(robots_url)
    parser.read()
    return parser.can_fetch(BOT_UA, url)


def fetch_page(url: str) -> str:
    if not robots_allows(url):
        raise PermissionError(f"robots.txt disallows this URL: {url}")

    # Keep this identity and session consistent for this crawl session.
    with requests.Session() as session:
        session.headers.update({"User-Agent": BOT_UA})
        response = session.get(url, timeout=30)
        response.raise_for_status()
        return response.text


if __name__ == "__main__":
    target = "https://example.org/"
    html = fetch_page(target)
    print(f"Fetched {len(html)} characters from {target}")

This small example is not a complete production crawler: RobotFileParser.read() does not provide sophisticated retry, caching, or policy handling. For a real crawl, decide how to handle robots.txt fetch failures in accordance with the applicable site policy, implement timeouts and backoff for transient errors, and avoid repeatedly downloading robots.txt. Do not silently treat a failed policy check as permission.

Using a browser UA header in other clients

For a permitted diagnostic request, a client can set a UA explicitly. This changes the header only; it does not reproduce the named browser’s network or rendering behavior.

curl -A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36' https://example.org/

In Python, the equivalent header-setting syntax is:

import requests

response = requests.get(
    "https://example.org/",
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36"},
    timeout=30,
)
response.raise_for_status()
print(response.status_code)

For Node.js, set the header on a supported runtime’s fetch request:

const response = await fetch('https://example.org/', {
  headers: {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36'
  }
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
console.log(await response.text());

Use a browser automation client when you actually need browser rendering or browser-consistent behavior. Avoid rotating these sample browser headers on requests made by a different client as a way to conceal its identity.

When a custom crawler string is better than a browser list

If your scraper is your own program rather than an actual browser, a transparent crawler identity is generally easier for site operators to interpret and for you to maintain. Use a stable product token and meaningful contact page, and apply robots.txt rules for that token. Do not label your bot as Googlebot or another named crawler unless it is that crawler. If you run several legitimate crawler variants, retain the same product token in each variant so the intended robots matching continues to work.

A browser profile list is most useful for authorized compatibility testing—such as checking whether a site serves different content to desktop and mobile browsers—or for sessions genuinely driven by those clients. For ordinary data collection, rotating browser disguises can make requests internally inconsistent while providing no guarantee of access.

Common problems and fixes

• The site still blocks the request. A UA is only one signal, and no universal success rate for rotation is established. Re-check permission, robots rules, request rate, cookies, client behavior, and the site’s access controls; do not respond by escalating identity rotation to evade a restriction.
• The page looks wrong or serves unexpected content. The UA may conflict with other headers, the cookie state, or the actual client. Choose a coherent profile and keep it through the session; if browser rendering is required, use a browser rather than a bare HTTP request.
• A previously working string now looks stale. Browser releases move on, and reduced UA formats do not provide all the precise OS and device values found in old lists. Refresh from supported clients and retire entries that no longer describe them.
• Robots rules appear not to match the crawler. Verify the product token in the request’s actual User-Agent and retain it in each crawler variant. RFC 9309 matching is case-insensitive, merges matching groups, and uses the wildcard group when no specific group matches.
• Rotation makes a crawl harder to debug. Log the selected profile, session or partition, target host, and response outcome. Keep selection stable long enough to reproduce the issue instead of choosing a random identity for every request.
• The example code fails before the page loads. Check that the target URL is valid and permitted, the network is available, and the installed Python package is present. A timeout or HTTP error is not solved by changing the UA; surface the error, apply suitable retry limits, and investigate the actual cause.

Performance, reliability, and maintenance trade-offs

A small verified set is cheaper to review and easier to debug than a large scraped list. More entries mean more opportunities for stale versions, inconsistent headers, and sessions that become difficult to reproduce. Rotation also does not make requests inherently faster or more reliable. The meaningful operational work is controlling request rate, using reasonable timeouts, preserving session state, handling transient failures, and caching when the site’s rules allow it.

There is no evidence here for a universal pool size, block-rate reduction, or success percentage. Results vary with the target’s controls, request rate, cookies, IP reputation, TLS and browser behavior, and site policy. If maintaining a local client pool becomes a poor fit for an authorized project, evaluate managed scraping infrastructure on its documented behavior and policy fit; Scrapy documentation mentions ban-avoidance services such as Zyte API, but availability and commercial terms should be checked with the provider.

Or skip the browser setup

If the job is to get a clean screenshot rather than to build a rotating scraping client, ScreenshotNeo is a screenshot API and MCP server. It accepts one GET request with a URL and returns a PNG, JPEG, WebP, or PDF. For its API options and response details, see the ScreenshotNeo documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

ScreenshotNeo accepts cookie or consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each of those steps can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing status. Its MCP server provides take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots, and every feature is on every plan.

Sign up for ScreenshotNeo’s free plan: 1,000 screenshots a month, no card required.

Sources and scope

The browser-format notes refer to MDN’s User-Agent reference material; the reduction and Client Hints notes refer to Chrome’s official User-Agent reduction documentation. The crawler-token and robots matching statements refer to RFC 9309, and the Scrapy identity note refers to Scrapy’s documentation. Version examples are not timeless: check the supported clients you actually operate before using them.

Frequently Asked Questions

Is there one User-Agent string that works best for every scraper?

No. The right identity depends on whether the request comes from a real browser profile or a crawler you operate, and on the target site’s rules. A stable, truthful identity is more maintainable than a universal-looking string.

Should I change User-Agents on every request?

Generally, no. Keep identity consistent through a logical session and change it only at a deliberate boundary. Per-request changes make behavior harder to reproduce and can conflict with session state.

Does a User-Agent rotation list guarantee fewer blocks?

No. A UA is one request attribute, and rotation does not override a site’s access controls or terms. No universal block-rate improvement is established.

Quick Recap

Bestseller No. 1

The Proxy Playbook: The Complete Guide to Proxy Servers: How to Source, Test, and Scale Residential, Mobile, and Datacenter Proxies for Web Scraping, ... Anonymous Browsing without Getting Blocked

$29.95

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.