Confirm the crawler is really the cause, relieve pressure at the layer that can act fastest, then replace the emergency fix with a crawler-by-crawler policy. A temporary 503 or 429 response protects the server in the short term. A robots.txt rule states policy to crawlers that honor it. A WAF or CDN rule enforces policy at the edge. They do different jobs, and the timing guidance below is Google’s for Googlebot. It should not be assumed for any other crawler.
Step 1: Confirm that a crawler is the problem
Start with evidence, not assumptions. Identify which agent is sending the requests and what its pattern looks like.
As an Amazon Associate I earn from qualifying purchases.
- Web-server logs: group requests by user-agent and IP range. Look at request volume over time, which paths are hit, and the status codes returned.
- CDN, WAF and bot-mitigation logs: if an edge layer sits in front of your origin, check its logs and bot events too. Requests blocked at the edge may never appear in origin logs, so origin logs alone can mislead you.
- Capacity indicators: line up the bursts against CPU, memory, database load, latency and error rates to see whether the crawler’s traffic matches the slowdown.
- Crawler reports: for Google, Search Console’s Crawl Stats report shows Googlebot activity. Google recommends monitoring for excessive Googlebot requests.
- 429 responses: OpenAI’s guidance for advertisers points to HTTP response codes (429 in particular), firewall/CDN logs, bot-mitigation events, throttling rules and traffic analytics as the places to look when a crawler is being rate limited. That advice is framed around ad landing-page access, but the checklist works generally.
Don’t trust the user-agent string alone
Anyone can send a user-agent header that claims to be a well-known bot. OpenAI publishes IP-range references for its crawlers and suggests combining user-agent identification with verified-bot programs where available, firewall allowlists and provider-level verification. Those IP lists and user-agent versions change, so read the operator’s current bot documentation before building an allowlist or blocklist. Not every burst is malicious either. A legitimate crawler can overload a small server simply by being fast.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteStep 2: Relieve the immediate pressure
If the crawler is Googlebot
Google Search Central’s crawling troubleshooting documentation (last updated 18 December 2025) says Googlebot has algorithms to prevent it from overwhelming your site with crawl requests. It still gives emergency steps for when you find it doing so. The advice is to temporarily return HTTP 503 or 429 to Googlebot while the server is near capacity, then stop once the crawl rate falls.
Google attaches two warnings, and they come from different pages:
- The troubleshooting page says that returning 503 or 429 for more than two days can cause the affected URLs to be dropped from Google’s index.
- The Search Console Crawl Stats help describes two quick options for an overcrawling Google agent: a robots.txt block, which can take up to a day to take effect, or dynamic 503/429 responses when you approach your serving limit. It warns that leaving either in place for more than two or three days can reduce Google’s crawling over the longer term.
Treat either as an incident measure with a removal date. Watch both crawl rate and host capacity while you recover.
If the crawler is anything else
Do not carry Google’s timing or indexing consequences over to other crawlers. None of the sources reviewed establishes a universal throttle value, retry schedule or recovery window for AI crawlers. Choose a temporary rate limit, challenge or block based on three things:
Free tools Windows power users keep installed
One-click scans. No signup required.
- whether you have verified the crawler’s identity,
- how much capacity you have,
- which controls your infrastructure actually offers.
Whether a given crawler backs off on 429 is something to check in its operator’s documentation and then confirm in your own logs.
Step 3: Pick the right control for the job
| Control | What it is | Speed and limits | Main risk |
|---|---|---|---|
| Temporary 503/429 | Enforced server response | Immediate relief at the server. Google documents this for Googlebot. | Over two days, Google says URLs can drop from its index. |
| robots.txt rule | Policy signal for crawlers that honor it | For Google, Crawl Stats help says up to a day to take effect. Google generally caches the file for up to 24 hours, possibly longer if it cannot refresh it. | Not enforcement. Left too long, it can hurt discovery. |
| WAF/CDN rule | Enforced edge action | Acts on matching requests before they reach the origin. Depends on your vendor’s features. | Misidentifying a crawler, or blocking one you wanted. |
robots.txt: policy, with sharp edges
robots.txt expresses which crawler may access which paths, provided the crawler respects the protocol. It is not a network-layer access control. Availability of the file matters too. Google’s robots.txt specification says a 4xx response other than 429 is treated as though no valid robots.txt exists, so a misconfigured file or a path that errors can silently remove your restrictions. Those response-code and caching details are Google’s behavior and shouldn’t be assumed for every crawler.
WAF and CDN rules: enforcement at the edge
When you need a block that does not depend on a crawler’s cooperation, or an exception more specific than robots.txt allows, enforce it at the edge. Cloudflare is one documented example. Its AI Crawl Control offers:
Rank #4
- a view of crawler activity,
- per-crawler allow or block choices, with block actions enforced through WAF custom rules,
- path-based exceptions through advanced rule customization in the WAF,
- a custom block response on paid plans.
These are Cloudflare product features, and other CDNs may differ. Plan availability can change, so check Cloudflare’s current documentation. Cloudflare also describes a pay-per-crawl option, including a charge action for successful crawl requests, but it identifies it as closed beta. Don’t plan around it as a generally available revenue source.
Step 4: Set a lasting policy, crawler by crawler
“AI crawler” covers several identities with different purposes. OpenAI’s current crawler documentation is a useful example of how much they can differ:
Best Value
- OAI-SearchBot: surfaces sites in ChatGPT search. Opting out means your site won’t be shown in ChatGPT search answers, though it may still appear as a navigational link.
- GPTBot: crawls content that may be used to train OpenAI’s generative AI foundation models. Disallowing it signals that your content should not be used for that training.
- OAI-AdsBot: visits pages submitted as ads to review landing pages. OpenAI says the data it collects is not used to train its foundation models.
- ChatGPT-User: fetches pages for certain user actions and is not an automatic web crawler. OpenAI says robots.txt rules may not apply because the visits are user-initiated.
This breakdown is specific to OpenAI. Other companies may use different categories or honor different directives, so read each operator’s documentation. The practical consequence is that a blanket block can cut off discovery you wanted, such as search visibility, while a narrow block can leave a heavy, unwanted crawler untouched.
Quick Recap
A workable policy usually does the following:
- Lists the crawlers you see in your logs and what each is for.
- Decides per crawler whether to allow it, restrict it to certain paths, rate limit it, or block it.
- Writes the allow and disallow choices in robots.txt, and enforces the ones that matter at the WAF or CDN.
- Protects expensive endpoints (search pages, faceted listings, dynamic APIs) first, because they cost the most per request.
- Reviews the logs after each change, and removes any temporary 503/429 or emergency block once crawl volume is back under control.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
