How to Write and Test robots.txt Rules for AI Crawlers (2026 Guide)

Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.

Write one UTF-8 plain-text file at /robots.txt on each host you need to control, then create separate User-agent groups for search, training, and user-requested retrieval bots. This lets you make an independent choice about ChatGPT search visibility, model-training use, and other AI access. It does not secure content: robots.txt is a voluntary crawler preference, and a non-compliant scraper can ignore it. Deploy the file on the live host, test representative URLs and failure states, and use authentication or bot-management controls for anything that must remain private.

What does robots.txt control for AI crawlers?

robots.txt is a machine-readable policy file that tells compliant crawlers which URL paths they may fetch. RFC 9309, the Internet standard published by the IETF in 2022, is explicit that these rules are not access authorization. Google likewise describes the file as instructions about which URLs a crawler can access, not as a security boundary.

#	Preview	Product	Price
1		SEO ENGINEERED: Search Visibility Architecture for Web Apps at Scale (Quality Control for AI-Built...	$2.99	Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

The policy applies only within the scope of the file’s protocol, host, and port. A file at https://www.example.com/robots.txt does not govern a different subdomain, a different hostname, or an HTTP endpoint. Serve a separate root file for every host whose paths need a policy.

• It can express: path-level Allow and Disallow preferences for named user-agent tokens, wildcards, and an optional sitemap location.
• It cannot guarantee: that an unknown scraper obeys, that a blocked URL is removed from an index, or that private data is protected from a direct request.
• Use security controls for secrets: put confidential material behind authentication and authorization, and add CDN, firewall, or bot-management controls where appropriate.

Because AI vendors separate their products by purpose, a single blanket rule can produce an unintended trade-off. The useful approach is to decide independently whether you want search discovery, training use, and user-initiated retrieval.

#1 Best Overall

SEO ENGINEERED: Search Visibility Architecture for Web Apps at Scale (Quality Control for AI-Built Apps)

$2.99

Buy on Amazon

Which AI crawler user-agent tokens should I control?

Use the token documented for the purpose you want to manage. Allowing a search crawler while disallowing a training crawler is technically possible because each group is evaluated independently.

Vendor names and purposes can change. Check the current OpenAI, Google, and Anthropic crawler documentation immediately before a policy change, especially if your decision depends on a distinction between training and user retrieval.

How do I create a valid robots.txt file?

Create a plain-text file named exactly robots.txt, encode it as UTF-8, and publish it at the lowercase root path. A word processor can insert formatting, smart punctuation, or metadata that makes a policy unreliable.

Use the correct location and response

1. For each host, place the file at https://host.example/robots.txt, not in a subdirectory.
2. Return it as plain text with the media type text/plain. Keep the response available without authentication.
3. Repeat the process for every protocol, hostname, port, and subdomain that needs a distinct policy.
4. Keep the file below Google’s 500 KiB limit. A very large generated file can be truncated or rejected even when its individual rules are correct.

Build groups with fields and rules

A group starts with one or more User-agent: lines and is followed by its Allow: and Disallow: rules. User-agent tokens are matched case-insensitively. Path matching starts at the beginning of the URL path and is normally case-sensitive.

The asterisk (*) matches zero or more characters, and a dollar sign ($) anchors a match to the end of the path. When several rules match, the most specific matching path wins. If an Allow and Disallow rule are equivalent in specificity, the Allow rule should win.

Start with a purpose-specific policy

# Keep ChatGPT search discoverability while opting GPTBot out of training use
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /private/
Disallow: /drafts/

# Google-Extended controls generative-AI use, not Google Search inclusion
User-agent: Google-Extended
Disallow: /licensed-content/

User-agent: ClaudeBot
Disallow: /training-sensitive/

Sitemap: https://www.example.com/sitemap.xml

Replace the example paths and sitemap host with your own. An Allow: / line is explicit, but it does not override a more specific disallow in the same applicable policy. A blank Disallow: value means no path is disallowed for that group.

How should I decide what to allow or block?

Make the decision by crawler purpose rather than by the word “AI.” A publisher that wants visibility in answer-oriented search may allow retrieval and search tokens while making a separate choice about training collection.

If ChatGPT search visibility matters

Give OAI-SearchBot the access needed to discover public pages you want surfaced. Blocking GPTBot does not, by itself, block OAI-SearchBot; OpenAI documents those settings as independent.

If you want to limit OpenAI training use

Add a GPTBot group with the paths you want excluded. Scope the rule narrowly when some sections are licensed or sensitive and other public pages remain eligible for discovery.

If you want Google Search but not generative-AI use

Use Google-Extended for the relevant paths. Google states that this token does not affect inclusion in Google Search and is not a Google Search ranking signal, so it is not a substitute for blocking the ordinary search crawler.

If you need to distinguish training from user requests

Keep ClaudeBot and Claude-User in separate groups, just as OpenAI separates GPTBot, OAI-SearchBot, and ChatGPT-User. A rule aimed at a training collector should not be assumed to govern a page fetched because a person asked an assistant to retrieve it.

When a rate preference is appropriate

Anthropic documents Crawl-delay as a non-standard extension for its bots. Do not assume that OpenAI, Google, or other crawlers recognize it; if request volume must be controlled reliably, use infrastructure-level rate limiting or a bot-management product.

How do matching, wildcards, and duplicate groups work?

Test matching as if the crawler were choosing the most specific path rule for its own token. The first rule in the file is not automatically dominant.

• Prefix matching: Disallow: /drafts/ covers that directory and paths below it.
• Wildcard matching: Disallow: /*.pdf$ targets paths ending in .pdf; the anchor prevents a later suffix from matching.
• Case: user-agent token comparison is case-insensitive, while path comparison normally preserves case. Treat /Reports/ and /reports/ as different URLs on a case-sensitive server.
• Specificity: a longer matching path takes precedence over a shorter one.
• Equal conflicts: when Allow and Disallow cover an equally specific path, Allow should win.
• Duplicate groups: avoid repeating the same token in separate groups. Combining its rules into one group makes the intended policy clear and avoids parser differences.

For example, to block an entire private area except one public feed, use a broad disallow and a more specific allow:

User-agent: GPTBot
Disallow: /private/
Allow: /private/public-feed.xml

Verify that your chosen parser treats the exception as more specific; then test the exact URL, its parent directory, and a similarly named path outside the directory.

How do I deploy robots.txt safely?

1. Draft in a plain-text editor. Use UTF-8, normal LF or CRLF line endings, and ordinary ASCII punctuation for field names and paths. Remove a byte-order mark (BOM) if your tooling inserts one.
2. Review the scope. Confirm the hostname, protocol, port, and subdomains represented by the deployment. A root file on the main domain cannot set policy for a separate subdomain.
3. Publish atomically. Upload the complete file rather than exposing a partially generated version. Keep a versioned copy so you can restore the last known-good policy.
4. Check the live response. Request the exact root URL from outside your network, inspect the status, content type, redirects, encoding, and body, and confirm that a CDN or firewall is not serving an error page.
5. Record the change. Note the deployment time and the intended policy for each token. Caches can make a correct change appear delayed.

Google’s documentation recommends direct browser verification, the Search Console robots.txt report where available, and Google’s open-source robots.txt library for local checks. These checks complement one another: a local parser catches logic errors, while a live fetch catches routing, CDN, and host-scope mistakes.

How do I test whether the deployed file works?

Test the deployed URL, not only the file on your laptop. Include every AI token you use and paths that exercise each rule type.

1. Fetch the file and inspect HTTP behavior

curl -iL --max-redirs 10 https://www.example.com/robots.txt

Look for a successful response, Content-Type: text/plain, the expected UTF-8 text, and the final host. A redirect chain should not silently move the policy to the wrong hostname. RFC 9309 says crawlers should follow at least five redirects; keeping the chain short is safer operationally.

2. Check syntax and encoding

• Confirm every directive uses a valid field:value form.
• Search for smart quotes, non-breaking spaces, invisible characters, and a BOM.
• Confirm line endings are consistent and the file remains under 500 KiB.
• Check that comments begin with # and that no generated HTML surrounds the text.

3. Test representative URL cases

For each token, test an allowed root page, a directly disallowed path, a nested path, a wildcard match, an end-anchored match, and an exception where Allow and Disallow conflict. Include case variants if your site treats URL case as significant. Run these cases through Google’s open-source robots.txt library or another standards-aware parser, then compare the result with your intended policy.

4. Use Google’s live reporting tools

Where your property is supported, inspect the Search Console robots.txt report after deployment. It can reveal that Google is reading a different host, an older version, or a malformed response even when a local file looks correct.

5. Verify crawler requests in logs

Review origin, CDN, and firewall logs for the documented user-agent tokens. Confirm the robots.txt request itself is not blocked by a bot rule, and watch whether requests continue to arrive for paths you intended to disallow. A log entry proves a request occurred; it does not prove a crawler is compliant, so use it alongside parser and vendor-tool checks.

What happens when robots.txt is missing, redirected, or broken?

Failure behavior differs from a normal, successfully parsed policy. Treat availability as an operational dependency and test it before relying on a restriction.

Missing or 4xx response

Under the Robots Exclusion Protocol, an unavailable policy can be interpreted as no restrictions, so a crawler may fetch URLs when the file is missing or returns an ordinary client-error response. Do not use an intentional 404 as a blocking strategy.

5xx or unreachable response

RFC 9309 requires complete disallow behavior while the file is unreachable, although crawler implementations can have their own retry and cache behavior. A transient origin outage can therefore create either a crawl pause or a delayed return to normal traffic. Monitor availability rather than assuming a failure has the same effect as a deliberate rule.

Redirect chains

Redirects to the correct policy can work, but long chains, cross-host redirects, loops, or a redirect to an HTML login page can leave a crawler without a usable policy. Test the final URL and every host involved.

Malformed directives

Unknown fields, broken encoding, or a missing separator can cause individual lines to be ignored. Keep directives simple, validate after every generated change, and isolate one user-agent group from another with clear blank lines and comments.

Stale caches

Google says it may cache robots.txt for up to 24 hours, or longer when a refresh fails. A correct deployment therefore may not change behavior immediately. Keep the old and new policies documented, and allow for cache expiry before interpreting logs as proof that a rule was ignored.

How can I prevent a robots.txt policy from exposing private content?

You cannot. Anyone can request a URL directly, and a non-compliant scraper can ignore the file. Put private material behind authentication and authorization, remove secrets from public responses, and use network or application controls when you need enforcement.

Robots.txt is still valuable for reducing accidental compliant crawling and for expressing publisher preferences. Treat it as a coordination signal, not as a permission check in application code.

What should I check before publishing a policy?

• The file is UTF-8 plain text at lowercase /robots.txt on every required host.
• The response is reachable without authentication and uses text/plain.
• Each AI purpose has the correct documented token and its own group.
• Search visibility and training choices are evaluated independently.
• Paths, wildcards, anchors, case, and equal-specificity conflicts have representative tests.
• Duplicate groups have been merged and comments explain unusual exceptions.
• The file is below 500 KiB and contains no BOM, smart punctuation, or generated markup.
• Redirect, 4xx, 5xx, network-failure, and cache-delay behavior has been considered.
• Search Console or an equivalent live report and Google’s open-source parser have been used where applicable.
• Logs show that the CDN, firewall, and origin permit robots.txt fetches.
• Authentication, authorization, and bot controls protect anything that must not be fetched.

The most maintainable policy is usually the smallest one that expresses your actual business decision: allow the crawlers that support the visibility you want, make a separate documented choice about training use, and test the live file after every change.

Frequently Asked Questions

Can I use one robots.txt file for all subdomains?

No. The policy is scoped to the exact protocol, host, and port that served it. Publish and test a root file on each subdomain whose paths need different or explicit instructions.

Will blocking an AI crawler remove pages already quoted or cached elsewhere?

A disallow rule governs future compliant fetching; it is not a takedown mechanism for copies, existing indexes, or third-party caches. Handle removal, licensing, or privacy issues through the relevant service and legal process.

Should I generate robots.txt dynamically for each request?

You can generate it, but deterministic output and fast availability matter more than personalization. Cache the result, validate the generated bytes, and keep a known-good fallback so a template or backend failure does not replace the file with an HTML error page.

How often should I retest a robots.txt policy?

Retest after changes to routing, CDNs, firewalls, URL structures, or vendor crawler documentation, not only after editing a rule. Periodic monitoring of the live response and access logs can catch an expired certificate, redirect change, or cache problem before it changes crawl behavior.

Quick Recap

Bestseller No. 1

SEO ENGINEERED: Search Visibility Architecture for Web Apps at Scale (Quality Control for AI-Built Apps)

$2.99

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

AI crawlersClaudeBotGoogle-ExtendedGPTBotRobots.txtTechnical SEO