Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallYes—but a robots.txt rule is a request to compliant crawlers, not a technical block. Site owners can use crawler-specific rules to express which content AI bots should avoid. To keep material private or reliably deny access, use authentication or server-, CDN-, or firewall-level controls instead.
What a crawler block does—and does not do
A site’s root robots.txt file can tell named crawlers which URLs they are asked not to access. The IETF’s RFC 9309, the Robots Exclusion Protocol, explicitly says these rules are not access authorization. A bot can ignore them, and a user-agent name can be imitated.
As an Amazon Associate I earn from qualifying purchases.
Google likewise describes robots.txt as a way to manage crawler access and traffic, not a way to keep a page private or guarantee its removal from search results. If a blocked URL is linked elsewhere, the URL may still appear in search results. Use access controls for confidentiality; use noindex where appropriate to control indexing, following the relevant search engine’s guidance.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Choose which AI use you want to allow
AI services may use different crawlers for different purposes, so a blanket rule can have unintended effects. OpenAI documents separate controls for search discovery and potential model-training crawling.
#1 Best Overall
| Crawler or control | Documented purpose | Effect to consider |
|---|---|---|
OAI-SearchBot |
OpenAI uses it to surface websites in ChatGPT search features. | OpenAI says opting out means a site will not appear in ChatGPT search answers, though it may still appear as a navigational link. |
GPTBot |
OpenAI says it crawls content that may be used to train generative AI foundation models. | Its control is independent of the OAI-SearchBot setting, so training-related crawling can be disallowed while search discovery remains allowed. |
ChatGPT-User |
OpenAI describes this as fetching pages in response to user actions, rather than automatic web crawling. | OpenAI says robots.txt may not apply to these user-initiated visits; an automatic-crawler rule is not a guarantee that every user-triggered request will be denied. |
Google-Extended |
Google says this standalone robots.txt product token controls whether content it crawls may be used to train future Gemini models and for certain grounding uses in Gemini Apps and Vertex AI. | Google says it does not affect inclusion in Google Search or act as a Search ranking signal. |
ClaudeBot |
Anthropic’s Help Center identifies it as a crawler and provides a robots.txt opt-out method. | Use Anthropic’s current instructions for the exact rule. |
These descriptions come from the operators’ documentation: OpenAI crawler information, Google’s robots.txt documentation, and Anthropic’s ClaudeBot help article. Operator behavior and documentation can change.
Write a targeted robots.txt rule
RFC 9309 places the file at the service root—for example, https://example.com/robots.txt—and defines user-agent groups with matching rules. A crawler-specific disallow rule can look like this:
Rank #2
- Protects against known exploits, malware and malicious websites; detects unknown attacks; identify thousands of applications
User-agent: GPTBot
Disallow: /This asks the named crawler not to access paths on the site. Add a separate group only if you also intend to block that crawler. For example, blocking OAI-SearchBot can remove the site from ChatGPT search answers; it is not required just because you want to disallow GPTBot.
For the relevant token and policy, check each operator’s current official documentation. The available evidence here does not establish Perplexity’s current crawler token, its role, or its robots.txt policy, so do not assume a particular rule will block its bots.
Rank #3
- Fortinet Web Application Firewall - virtual appliance for all supported platforms. Supports up to 1 x vCPU core
- Fortinet HW FWB-VM01
- Manufacturer Part: FWB-VM01
When a rule is not enough
If the content must be inaccessible, require authentication or deny requests through server, CDN, or firewall controls. Do not rely on a bot’s declared user-agent as the only security boundary, because it can be copied. When using infrastructure rules, check the current documentation for your hosting, CDN, or WAF provider.
After publishing a robots.txt change, check that the root file is reachable and contains the intended user-agent group and path rule. If your goal is enforced denial, also test the actual request path and response and verify that server and network-edge rules do not permit access. This is operational guidance derived from the protocol’s distinction between crawler instructions and authorization, not a guarantee provided by robots.txt.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Allow search discovery while limiting training-related crawling
For OpenAI, the documented controls are independent: a site can allow OAI-SearchBot for ChatGPT search discovery while disallowing GPTBot for potential training-related crawling. Google-Extended is a separate control and, according to Google, does not determine Google Search inclusion or ranking. These choices apply to the documented crawlers and uses; they do not make public pages private or control every possible user-triggered visit.
OpenAI says robots.txt changes can take about 24 hours to affect its search systems. That is OpenAI’s stated operational timing, not an immediate guarantee or a general propagation time for other operators.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
