AI crawlers can slow a WordPress site when their requests bypass the cache and trigger PHP, plugins, database queries, logins, searches or APIs. The practical fix is not a blanket ban: measure which agents and URLs consume resources, cache what can be cached, filter unwanted automation at the edge, and keep verified search and operational bots working. Use robots.txt to state your preference, but enforce it with CDN or WAF rules because compliant instructions alone cannot protect a server.
Antihost offers managed WordPress hosting with free migration, one-click staging and a 14-day free trial.
Why AI bot requests become expensive on WordPress
A cached article can often be served without starting WordPress. An uncached request is different: it may allocate a PHP worker, load the theme and plugins, run several database queries, create sessions, and generate a response before the connection closes. A crawler that opens many concurrent connections turns that per-request work into sustained pressure on the same limited pools used by human visitors.
As an Amazon Associate I earn from qualifying purchases.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsThe resulting symptoms are familiar: rising time to first byte, exhausted PHP workers, high CPU or memory use, too many database connections, bandwidth spikes, 5xx responses, host throttling, or resource-limit notices. A bot is not automatically harmful, however. A verified search crawler fetching cacheable pages may cost less than an unverified script repeatedly requesting search, login or API endpoints. Measure the mechanism in logs rather than judging traffic by its user-agent string alone.
How large is the AI-crawler problem?
Cloudflare’s 2025 Radar review found that AI bots represented an average 4.2% of HTML requests, with monthly values ranging from 2.4% to 6.4%. Googlebot alone averaged 4.5% and reached 11% in late April. These are aggregate measurements, not a prediction for every WordPress site: your percentage depends on audience, content, cache policy, geography and which bots can reach the origin.
#1 Best Overall
Cloudflare also compared referrals with crawler activity in a June 2025 analysis. Google crawled about 14 times per referral, while OpenAI was estimated at 1,700 crawls per referral and Anthropic at 73,000. Cloudflare cautions that these are estimates and that referrals from native applications can be unattributed. A high crawl-to-referral ratio is a reason to inspect cost and intent, not proof that every request should be blocked.
Diagnose the requests before changing rules
Establish a baseline and preserve a rollback path. A short, structured investigation prevents you from breaking search visibility or an integration while chasing a harmless cache hit.
- Correlate resource use with traffic. Compare origin CPU, memory, PHP-worker utilization, database connections, response time and 4xx/5xx rates with request spikes. Note whether the spike occurs at the edge, at the origin, or both.
- Group access logs. Segment by user agent, source IP or ASN, URL path, HTTP method, status code, response time and cache hit or miss. Look for repeated requests from the same identity and for bursts that line up with worker exhaustion.
- Classify the automation. Separate verified search crawlers, AI training crawlers, AI-search or user-action crawlers, uptime monitors, authenticated integrations and clearly abusive scripts. Verification should rely on your CDN or server’s documented method, not only on a self-declared user-agent.
- Find expensive paths. Check uncached home and archive pages, internal search, login screens, XML-RPC, REST endpoints, feeds, and WooCommerce cart or checkout URLs. Record which paths miss cache and which return errors.
- Set success criteria. Write down normal origin load, median and high-percentile response time, search-console crawl signals, index coverage and real-user error rates. These become your before-and-after comparison.
| Path or request type | Why it can consume resources | What to inspect |
|---|---|---|
| Public articles and archives | Usually inexpensive when a full-page cache is warm; costly when every request invokes PHP. | Cache status, response time and regeneration frequency. |
| Internal search | Often builds database queries for each unique query and is difficult to cache safely. | Query patterns, request rate and database time. |
| Login and XML-RPC | Authentication and write-capable handlers can bypass page caching and invite abuse. | POST volume, failed logins, source reputation and response codes. |
| REST or custom APIs | May execute plugin code or return personalized data on every call. | Endpoint, method, authentication state and cache headers. |
| Cart and checkout | Sessions and inventory calculations make these requests dynamic by design. | Authenticated traffic, false positives and origin latency. |
Controls, from least to most forceful
| Control | Where it acts | Selectivity | Strength and trade-off |
|---|---|---|---|
robots.txt |
Client-side preference | User-agent and path | Easy and transparent, but voluntary. Cloudflare describes it as an “honor system.” |
| Full-page and object caching | Edge and origin cache | Cacheability, URL and headers | Reduces PHP and database work for repeatable responses; cannot safely cache every personalized path. |
| CDN/WAF bot rules | Edge, before WordPress | Verified status, bot score, path, IP or ASN and user-agent | Stops work before it reaches the origin; requires careful exceptions and a plan that exposes the needed controls. |
| Rate limiting | Usually edge, sometimes origin | Matching expression plus counting key and path | Controls bursts without a total ban, but counters can lag and excess requests may briefly pass through. |
| Origin firewall | Server perimeter | IP, network and port | Useful as a backstop; filtering here still consumes some network and connection resources, and must allow your proxy ranges when a CDN is in front. |
Step 1: make the origin cheaper before blocking
Cache stable HTML
Enable full-page caching for public posts, pages and archives, then verify that repeated anonymous requests are served from cache. Exclude login, cart, checkout, account pages and any response containing private or session-specific data. Set sensible cache lifetimes and purge rules so publishing changes does not leave stale pages indefinitely.
Use object caching and database hygiene
An object cache can keep repeated WordPress queries out of the database. Remove or tune plugins that issue expensive queries, and make sure persistent cache connections are sized for your host. The goal is to reduce the work each request performs; blocking a bot is less effective if ordinary visitors still trigger an unnecessarily heavy page build.
Put a reverse proxy in front
A CDN or reverse proxy can answer cacheable requests and reject unwanted traffic at the edge. Where your architecture supports it, restrict direct origin access to the proxy’s published IP ranges. Test health checks, administrative access and deployment callbacks before tightening the firewall, and keep an emergency origin route for recovery.
Step 2: allow useful crawlers and challenge the rest
Start with verified search crawlers such as Googlebot and Bingbot when search visibility matters, and allow monitoring services and authenticated integrations that your operations depend on. Then apply progressively stronger actions to unknown or unwanted automation: observe, log, challenge, throttle or block.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Rank #3
Cloudflare documents a bot score from 1 to 99. Lower scores indicate scripts or automated agents; higher scores indicate normal browsers. Its examples allow verified bots, block score-1 traffic and apply a managed challenge to scores 2–29. Treat those ranges as a starting policy, not a universal truth. Review challenge success and false positives for your audience before expanding a rule.
Use path-specific policies
Protect /login/, /wp-login.php, /xmlrpc.php, internal search and expensive APIs more aggressively than public, cached articles. A rule can require a challenge for anonymous automation on a sensitive path while leaving normal article requests available. Explicitly exempt legitimate API clients, uptime monitors and authenticated integrations, and document each exemption so a later cleanup does not disable it.
Step 3: add rate limits without expecting perfect precision
A rate rule consists of a matching expression, counting characteristics, a period, a request threshold and a mitigation duration. Begin with narrow rules around the paths identified in your logs. For example, count requests to login or search per source identity over a short period, then challenge or temporarily block only after the observed threshold is exceeded. Separate authenticated users from anonymous traffic where your platform can do so safely.
Rank #4
Rate limits reduce bursts; they do not create an exact allowance. Cloudflare warns that its rules are “not designed to allow a precise number of requests to reach your origin server.” Counter updates can lag by seconds, so some excess requests may reach the origin before mitigation. Watch origin load and error rates after deployment, then adjust the period, threshold, counting key and mitigation duration. Do not rate-limit verified search bots casually: Cloudflare warns that doing so can affect SEO.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Step 4: publish a robots.txt policy, knowing its limit
Use robots.txt to communicate which crawlers and paths you prefer to exclude. A minimal example is:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
Replace the scope with the policy you actually want; a site that permits indexing of public documentation may disallow only administrative or high-cost paths. Keep the file at the site root, check its syntax, and confirm the response is publicly reachable. This instruction works only when a crawler chooses to comply. As Cloudflare puts it, “robots.txt is an honor system.” A bot that ignores the file still needs an edge, WAF or origin control.
Best Value
Protect SEO, users and business-critical automation
- Keep verified search access. Allow the search crawlers you rely on, and monitor crawl statistics and index coverage after each rule change.
- Do not blanket-block all automation. Monitoring, accessibility, feeds, partner APIs and security checks can be operationally important even when they are not browsers.
- Protect dynamic journeys from false positives. Test login, checkout, account pages and API calls from representative networks, devices and authenticated states.
- Compare real-user signals. Track analytics referrals, browser error reports, conversion steps and support complaints alongside server metrics. A lower bot count is not a success if legitimate users are challenged.
- Change one policy layer at a time. Deploy cache changes, bot actions and rate rules separately when possible, with timestamps and an immediate rollback procedure.
When a rule causes trouble
Search traffic or indexing falls
Check whether a verified crawler was misclassified, whether its IP range is allowed, and whether a rate rule matches its requests. Review search-console crawl behavior and index coverage against your baseline, then narrow the rule or add a verified-bot exception.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Origin load remains high after blocking
Confirm that the rule executes at the edge and that traffic is not reaching an unprotected origin hostname. Inspect cache-miss paths, POST requests and API calls; blocking article crawls cannot solve a separate login or XML-RPC flood. Verify that proxy-to-origin connections are not bypassing the intended firewall policy.
Legitimate visitors see challenges or 403 errors
Identify the matching dimension: bot score, ASN, IP reputation, user-agent, path or request rate. Remove broad network blocks, lower the sensitivity for public pages, and add narrowly scoped exceptions for trusted monitors and integrations. Re-test from mobile and corporate networks, not only from your own connection.
Rate limiting appears inconsistent
Account for counter lag and distributed clients. A source-IP key may miss a crawler spread across many addresses, while a broad ASN key can catch legitimate users. Use edge analytics to compare the rule’s matches with origin requests and tune gradually.
A maintenance routine that keeps controls accurate
- Review edge analytics and origin logs weekly for new user agents, paths, cache misses, response codes and burst patterns.
- Recheck exceptions whenever you add a plugin, API integration, monitoring service or checkout feature.
- Compare bot volume with CPU, PHP workers, database connections and bandwidth rather than relying on request count alone.
- After meaningful changes, watch real-user latency, 4xx/5xx rates, search-console signals and conversion paths for at least one normal traffic cycle.
- Keep dated copies of firewall, bot and rate-limit policies so you can restore the last known-good configuration quickly.
The durable strategy is layered: make repeatable pages cheap to serve, identify which automation is costly, enforce selective controls before requests reach WordPress, and preserve verified search and operational access. AI-crawler traffic is measurable, but its risk depends more on cache misses and path behavior than on the label in a user-agent string.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
