The practical answer: add separate User-agent groups for the AI crawlers you want to restrict in WordPress’s virtual /robots.txt, then verify the production file. For example, you can block training crawlers while leaving ordinary search-engine rules unchanged:
Antihost offers managed WordPress hosting with free migration, one-click staging and a 14-day free trial.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /This is a crawler preference, not a security wall. A bot that ignores robots.txt requires enforcement at your CDN, web application firewall (WAF), server, or authentication layer. The key to avoiding accidental search traffic loss is to treat training crawlers, AI search bots, and user-triggered assistants as separate policy targets.
As an Amazon Associate I earn from qualifying purchases.
Free tools Windows power users keep installed
One-click scans. No signup required.
What blocking an AI crawler actually controls
WordPress normally serves a virtual robots.txt response rather than a file you must upload to the site’s document root. The response tells named user agents which URL paths they should not crawl. A rule such as Disallow: / asks the named crawler not to request any path on the site.
That instruction does not remove pages already collected, prevent a human from visiting a public URL, or authenticate requests. It also does not automatically cover every bot operated by the same company. Each identity needs its own policy decision.
#1 Best Overall
Training crawlers, search bots, and assistants are different
Cloudflare’s bot reference, updated April 23, 2026, lists separate identities including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-CloudVertexBot, Bytespider, CCBot, Meta-ExternalAgent, and Amazonbot. Blocking one name does not automatically block the others.
| Identity category | Policy question | Typical robots.txt treatment |
|---|---|---|
| Model-training crawler | Should this agent collect material for future model datasets? | Add a dedicated group, such as GPTBot or ClaudeBot, when you want to opt out. |
| AI search crawler | Do you want pages considered for an AI search index or answer engine? | Decide separately for names such as OAI-SearchBot, Claude-SearchBot, or PerplexityBot. |
| User-triggered assistant | Should an assistant retrieve a page when a user explicitly asks for it? | Evaluate identities such as ChatGPT-User and Claude-User independently. |
| Ordinary search engine | Should conventional search indexing continue? | Keep existing Google and other search-engine rules unless changing indexing is intentional. |
Anthropic’s official help center says site owners can block ClaudeBot for the whole website with robots.txt: “When a site restricts ClaudeBot access, it signals that the site’s future materials should be excluded from our AI model training datasets.” That statement concerns ClaudeBot; it is not a blanket rule for every Anthropic identity.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Choose the scope before editing WordPress
Write down the result you want before adding a directive. A narrow policy can block training collection while allowing selected search or retrieval traffic. A broad policy can target every listed AI identity, but it may reduce discovery in AI-powered search and assistant responses.
Three common policies
- Training-only opt-out: block known training identities such as
GPTBotandClaudeBot, then decide separately about search and user agents. - AI-discovery opt-out: block training, AI search, and assistant identities. This is more restrictive and can reduce appearances in AI answer products.
- Selective allowance: block training agents but allow one or more search agents after considering referral value, licensing, and editorial goals.
Do not use a generic “AI bot” assumption as your only rule. New names can appear, user-agent strings can change, and a bot can misrepresent its identity. Maintain a list of the identities you intentionally target and review it after hosting, plugin, or CDN changes.
Method 1: Add user-agent rules to WordPress’s virtual robots.txt
Manual rules give you precise scope and do not require a crawler-blocking plugin. They are appropriate when you have a short, known list and can review the file after edits.
Step 1: Inspect the current production response
- Open the site’s public robots URL in a browser, for example https://example.com/robots.txt. Replace
example.comwith your own production host when testing. - Look for existing groups, sitemap lines, and search-engine directives. Save a copy before changing anything.
- Check whether the response is generated by WordPress, a security plugin, your host, or a CDN. Two systems writing different versions can make the served result differ from the editor view.
Step 2: Add one complete group per crawler
Use the exact user-agent token and a separate group for each identity you want to restrict:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /Only include the groups that match your policy. If you intend to block training crawlers but not AI search, omit the search identities. Conversely, adding OAI-SearchBot is necessary if your policy also covers that identity; blocking GPTBot alone does not do so.
Step 3: Preserve unrelated directives
Do not replace an existing search-engine section merely to add AI rules. Keep legitimate sitemap declarations and ordinary search rules unless you have explicitly decided to change them. A careless global Disallow: / under a broad or wildcard group can stop search crawlers as well as AI agents.
Step 4: Publish and verify the served file
- Save the WordPress setting or file that controls the virtual response.
- Request the live URL from a shell, not only from an administrator preview:
curl -sSL https://example.com/robots.txt. - Confirm that each intended
User-agentgroup andDisallow: /line appears in the response. - Check the response from a network outside your office or VPN if your CDN varies by location.
A successful save in WordPress does not prove that visitors receive the new content. Server caches, page-cache plugins, and CDNs can continue serving an older copy. Purge the relevant cache, wait for its documented refresh interval, and request the production URL again.
Method 2: Use a maintained WordPress crawler-blocking plugin
The WordPress.org documentation for the Block AI Crawlers plugin says it can generate and maintain named rules in WordPress’s virtual robots.txt. Its documented coverage includes crawler identities associated with OpenAI, Google, Meta, Anthropic, Perplexity, Apple, and Amazon.
When a plugin is useful
- You want a switchable list instead of editing text manually.
- Several people manage the site and need a visible settings screen.
- You want the list of named identities maintained as vendors add or rename crawlers.
Plugin installation and review checklist
- Install the plugin from the WordPress administration area and confirm that it is actively maintained for your WordPress version.
- Review every proposed crawler before enabling a “block all” or similarly broad option. The names may include training, search, and assistant agents with different consequences.
- Activate only the groups that match your written policy.
- Open the site’s public
/robots.txtand verify that the plugin’s generated rules are actually present. - Compare the live response with the plugin screen after clearing any page, object, host, or CDN cache.
The plugin documentation warns that the generated file may be cached by a server, plugin, or CDN. Treat the served response as authoritative. If a plugin update changes the list, repeat the identity and search-visibility review rather than assuming the old policy remains unchanged.
Method 3: Enforce the policy at a CDN or WAF
Use edge enforcement when compliance with robots.txt is insufficient or when you need to reduce requests before they reach WordPress. Cloudflare documents an auto-updating “Block AI bots” managed rule, a “Managed robots.txt” feature that prepends AI crawler directives, and WAF custom rules and bot-management fields for edge enforcement.
Managed robots.txt
A managed robots response can keep directives current and place them ahead of WordPress-generated content. It remains advisory: a compliant crawler will read it, but a non-compliant request can still arrive at the origin.
WAF and bot-management rules
Edge rules can challenge, rate-limit, or block requests that match a known crawler identity or bot signal. Their scope is broader than a text file and can protect server resources, but an overly broad rule can block legitimate users or search traffic. Start with a narrow identity match, inspect logs, and expand only when false positives are understood.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →When to combine methods
A practical layered configuration is to publish a clear robots policy for compliant crawlers and enforce the same named identities at the edge. Keep the lists synchronized. If the CDN blocks a crawler but WordPress advertises access, or the reverse, your documentation and troubleshooting become harder and a future configuration change can reopen traffic.
How to test each crawler rule
Testing should answer two different questions: what policy is published, and what traffic is actually allowed.
Check the published policy
- Fetch
https://example.com/robots.txtwithcurl -sSLafter every edit. - Search the response for the exact spelling and capitalization of every intended identity.
- Confirm that a new directive did not remove an existing sitemap or search-engine group.
- Repeat the request after purging caches and after changing plugins, hosting, or CDN settings.
Check enforcement separately
- Review CDN, WAF, and origin logs for requests identifying as the targeted user agent.
- Confirm that a rule is matching the intended identity rather than every automated request.
- Look for false positives involving accessibility tools, monitoring services, feed readers, or ordinary search crawlers.
- Test from the production hostname, because a staging hostname may use a different robots response and security policy.
A user-agent header can be spoofed. Logs therefore help you measure ordinary compliance, but they cannot prove the identity of every automated client. For high-value or private material, use authentication or access controls instead of relying on a name in a header.
What robots.txt cannot do
robots.txt expresses a site owner’s preference; it is not cryptographic protection or a network access barrier. Cloudflare’s crawler analysis notes that compliance is voluntary. A crawler that ignores the directive can still request public pages unless another control stops it.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11- It cannot protect private content: put confidential material behind authentication and authorization.
- It cannot erase prior collection: blocking future crawling does not recall copies already obtained.
- It cannot guarantee identity: a malicious client can send a different user-agent string.
- It cannot replace rate limiting: a compliant bot may still make many requests before stopping, depending on its crawl behavior.
For non-compliant or abusive traffic, consider CDN/WAF blocking, server-level deny rules, rate limits, signed URLs, authentication, or removing the material from public access. Choose the least disruptive control that solves the actual problem.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Costs and traffic trade-offs to evaluate
Blocking can reduce unwanted collection, but it can also reduce referrals or visibility in AI-powered discovery. Cloudflare reported an OpenAI crawl-to-referral ratio of 1,700:1 and an Anthropic crawl-to-referral ratio of 73,000:1 in its June 2025 analysis. Those are Cloudflare’s measurements for that analysis, not a universal rate for every publisher or period.
Cloudflare also examined 3,816 top-domain robots.txt files in June 2025 and found 546—about 14% of that sample—with allow or disallow directives targeting AI bots. This is a result from Cloudflare’s stated sample and method, not a statistic about the entire web.
| Approach | Scope | Enforcement | Maintenance | Main failure modes |
|---|---|---|---|---|
| Manual WordPress rules | Precisely named crawlers | Advisory | You maintain names and syntax | Typos, stale identities, cache delay, accidental search blocking |
| Block AI Crawlers plugin | Maintained named list, depending on its current configuration | Advisory through virtual robots.txt |
Plugin supplies list updates; you review scope | Generated file cached, broad selection, plugin conflicts |
| CDN managed robots.txt | Vendor-defined AI groups and directives | Advisory unless paired with edge rules | Vendor updates signatures or directives | Unexpected precedence, cache behavior, policy drift |
| CDN/WAF custom enforcement | Requests matching selected bot signals | Challenge, rate-limit, or block at the edge | You maintain conditions and exceptions | Spoofed headers, false positives, blocking legitimate traffic |
Use the narrowest option that matches your objective. If your only goal is to opt out of model training, blocking every AI search and assistant identity may sacrifice discovery you meant to keep. If your priority is origin protection from non-compliant scraping, an advisory file alone is unlikely to be enough.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Maintenance plan for a WordPress site
- Keep a policy inventory: record each blocked, allowed, and undecided identity, with the reason for the decision.
- Review crawler references: compare your inventory with current bot documentation when vendors introduce new names.
- Recheck search rules: after every plugin, hosting, or CDN change, confirm that ordinary search-engine directives remain intact.
- Monitor logs: look for targeted identities, unexpected request volume, and legitimate clients caught by edge rules.
- Test cache behavior: request the public file after purges and configuration changes, not just from an authenticated WordPress session.
- Escalate when necessary: use authentication, rate limiting, or WAF controls for clients that disregard the published preference.
Recommended decision path
For most WordPress publishers, start with explicit groups for the training crawlers you have decided to exclude. Leave search and user-triggered identities as separate choices, and preserve ordinary search-engine rules. Verify the production response with curl, clear caches, and inspect logs.
Add a maintained plugin when the list is broad or several administrators need a switchable interface. Add CDN/WAF enforcement when requests must be stopped rather than merely discouraged. In every configuration, document the exact identities covered and retest after infrastructure changes. This approach blocks the traffic you actually intend to block without treating “AI crawler” as one indistinguishable category.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
