Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteRobots.txt tells compliant web crawlers which parts of a site they should avoid; it does not secure those pages or guarantee they stay out of search results. Use it to guide crawler requests, use a crawlable noindex directive to keep an accessible page out of Google Search, and use server-side access controls to protect private content.
What robots.txt does
Robots.txt is a plain-text set of instructions for automated clients, commonly called crawlers or bots. It is normally published as /robots.txt at the root of the relevant site authority. The file can ask crawlers that follow the protocol to avoid specified paths, which can help manage crawl traffic. It does not make a path inaccessible.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
Advanced Robots.txt Generator Manual | $32.46 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
The Internet Engineering Task Force’s September 2022 Standards Track specification, RFC 9309, Robots Exclusion Protocol, states: “These rules are not a form of access authorization.” The standard also notes that paths listed in robots.txt are publicly exposed and recommends valid application-layer security, such as HTTP authentication, when access must be controlled.
How rules select crawlers and paths
A robots.txt file uses groups of directives. User-agent names the crawler group; Disallow and Allow specify paths for that group. A wildcard group, User-agent: *, applies when no more specific group is selected. RFC 9309 says crawlers use the most specific matching path rule; equivalent Allow and Disallow matches should resolve in favor of allowing access. Individual crawler implementations may differ in how they interpret or apply rules.
#1 Best Overall
User-agent: ExampleBot
Disallow: /private-looking/
User-agent: *
Allow: /In this example, the first group asks ExampleBot to avoid paths under /private-looking/; the wildcard group allows other paths. The path name is only a label. It does not make the content private, and listing a sensitive-looking path can reveal that it exists.
Rules apply only within the scope of the robots.txt file’s protocol, host, and port. As Google explains in its robots.txt introduction, a file on one host does not automatically govern a sibling subdomain, a different protocol such as HTTP instead of HTTPS, or a different port. Publish the applicable file for each authority whose crawler requests you want to guide.
Does robots.txt stop bots?
It can guide crawlers that choose to honor its rules, but it cannot force every bot to comply. Google says crawler behavior is up to the crawler, and RFC 9309 is a protocol for crawler instructions—not an access-control system. A disallowed URL may still be visited by noncompliant clients, and people can still request it unless the server restricts access.
Free tools Windows power users keep installed
One-click scans. No signup required.
Does robots.txt keep a page out of Google?
Not reliably. A Disallow rule can stop Googlebot from fetching a page, but Google may discover the URL through links and show the URL in Search without crawling its content. That can mean the result lacks a page-derived title or snippet; disallowing a path is not a dependable way to remove its URL from search results.
For an accessible page that should not appear in Google Search, let Googlebot fetch the page and serve a supported noindex directive. Google supports noindex in a page’s meta tag or in an X-Robots-Tag HTTP response header. A robots.txt block can prevent Google from seeing either directive. Google does not support noindex as a robots.txt rule. See Google’s guide to blocking indexing.
Choose the mechanism for the outcome you need
| Goal | Mechanism | Key limitation |
|---|---|---|
| Reduce requests from compliant crawlers to selected paths | Robots.txt rules | Crawlers can ignore the instructions; they do not restrict human or unauthorized access. RFC 9309 |
| Keep an accessible page out of Google Search | A crawlable noindex meta tag or X-Robots-Tag header |
Google must be able to fetch the page to read the directive. Google Search Central |
| Keep private content inaccessible | Server-side authentication or another valid access control | Do not rely on a robots.txt rule or publish secrets in a public path. RFC 9309 |
What if a crawler cannot fetch robots.txt?
The protocol’s guidance and Google’s behavior are related but not identical. Under RFC 9309, a 4xx response means the file is unavailable and may allow a crawler to access resources. If server or network errors make the file unreachable, the standard requires a crawler to assume complete disallow. It recommends that cached rules generally not be used for more than 24 hours unless the file remains unreachable. These are protocol rules, not a guarantee that all crawlers handle failures the same way.
Google documents its own operational handling in its robots.txt specifications: it treats most 4xx responses other than 429 as if no crawl restrictions exist. With 5xx errors, Google initially pauses crawling and retries; it can then use a cached file for a period. Google generally caches robots.txt for up to 24 hours, but may keep a cached copy longer if it cannot refresh the file.
Protocol limits worth knowing
RFC 9309 requires crawlers to support parsing at least 500 KiB of a robots.txt file. It also recommends a cache period of no more than 24 hours unless the file is unreachable. Google separately documents a 500 KiB limit and its own caching behavior in its implementation guidance. These are technical limits and recommendations, not measures of how many sites or bots use the protocol.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
