Limit the costly or abusive request pattern—not every automated visitor. Use access logs and Google Search Console’s Crawl Stats to identify what is generating the load, verify legitimate Googlebot traffic, then apply a measured rate limit to the relevant endpoint or action at your application or CDN/WAF. A site-wide throttle or a rule based only on a user-agent string can block search crawling along with scrapers.
1. Find out what is causing the load
Before changing a rule, compare your server or CDN access logs with Google Search Console’s Crawl Stats report. Look for the client, requested host and paths, response codes, request frequency, and whether traffic is concentrated on expensive endpoints. Google recommends checking logs and Crawl Stats when crawl activity changes; a large new section, newly unblocked pages, or many ad targets can also increase crawling. See Google’s crawl-rate guidance.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
Bot Traffic in Practice: Architecture, Detection, and Operations for Bot Defense | $9.99 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
Separate a genuine rise in Googlebot activity from requests by other crawlers or clients. Check whether the load comes from many new URLs, query-string variations that create repeated requests, a costly API route, or a broader capacity problem. A short burst is not enough on its own to establish abuse: Google says most sites should not see Googlebot access more than once every few seconds on average, while brief bursts can appear higher because of delays.
2. Verify search crawlers before writing exclusions
A request claiming to be Googlebot is not necessarily from Google. Do not trust the user-agent string by itself; verify crawler identity using Google’s verification methods before exempting or blocking traffic. Then make sure the relevant CDN/WAF and origin rules preserve verified Google crawler requests. Cloudflare likewise warns against blocking Google crawler IPs or user agents, or rate-limiting Google crawler traffic in error; see its crawl troubleshooting guidance.
Do not assume robots.txt is a complete traffic firewall. It is a crawl instruction, not proof of a client’s identity or a substitute for targeted edge or application enforcement. Google also notes that mobile and desktop Googlebot use the same product token in robots.txt, so that file cannot target those two crawler types separately.
3. Rate-limit the expensive behavior, not the whole site
Choose a rule that matches the work consuming resources. If price lookups are costly, limit that action; if downloads are being repeated, scope the rule to the download path or resource. Cloudflare’s rate-limiting examples show how policies can be scoped using an endpoint, session, path, header, or another request characteristic. Where available, use observed traffic or API Discovery to understand normal usage before setting a threshold.
- Choose a counting key that fits the request. An authenticated API may be better counted by token or session; a per-resource download may be better counted by path. Avoid a single shared limit that lets one client consume the allowance for everyone.
- Set a threshold from your own traffic and capacity. There is no universally safe request rate in the official guidance. Account for normal peaks and the cost of the endpoint, then adjust based on logs and origin health.
- Choose the least disruptive action that works. A rate limit, block, or challenge can have different effects; apply the policy to the abusive behavior and avoid sending verified search crawlers an unintended challenge or limit.
Cloudflare’s AI crawler controls also distinguish Search, Agent, and Training behaviors. Where those controls are available in your account, choosing policy by purpose is more precise than treating every “AI bot” label as the same kind of traffic. Its documentation describes AI Crawl Control allow/block actions; Pay per crawl is identified there as closed beta, not a generally available feature. See Cloudflare’s bot documentation.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →4. Use Google’s crawl controls only when Googlebot is the problem
If verified Googlebot itself is overwhelming the site, Google documents temporary overload responses as emergency relief. Returning 500, 503, or 429 responses can reduce crawling across the hostname, not just for the URL returning the error, so this is not a routine scraper filter. Google says to use the approach only briefly—hours or, at most, around one or two days—and warns that sustained errors can affect how URLs appear in Google products; repeated errors on a URL for multiple days may cause it to drop from the index.
Google’s crawl-rate guidance also provides a special request path if returning errors is infeasible. Evaluation may take several days. For a temporary robots.txt block, Google Search Console’s troubleshooting advice says the change can take up to a day to take effect and warns against leaving it in place too long; remove temporary blocks or responses after the crawl rate adapts, typically within two or three days.
5. Confirm the rule helps without disrupting indexing
After deploying a change, watch origin load and availability alongside status codes, access logs, and Crawl Stats. Check that expensive or abusive requests have fallen and that verified Google crawlers are not receiving unintended rate limits, challenges, or errors. Cloudflare recommends monitoring performance and availability and checking the original client IP in logs. If important pages stop being crawled or verified crawlers are affected, narrow or roll back the rule.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errors
