The key difference between an AI crawler and a search crawler is what the operator uses it for—not simply whether it is labelled “AI.” A provider may use separate bots or controls for search discovery, model development, and page retrieval requested by an individual user. To manage access, decide which of those outcomes you want, apply the provider-specific rule, then verify requests using both logs and official IP information.
What makes an AI crawler different from a search crawler?
“AI crawler” is not one standard category. Some crawlers collect pages for possible model development; others help a search product find and link to pages. A third type retrieves a page after a user asks a question. Those purposes can belong to separate agents at the same provider, and their controls may not work alike.
As an Amazon Associate I earn from qualifying purchases.
For example, OpenAI documents OAI-SearchBot for surfacing sites in ChatGPT search features, GPTBot for crawling content that may be used in model training, and ChatGPT-User for certain user actions. Anthropic describes corresponding search, potential training-data collection, and user-requested retrieval agents. Perplexity distinguishes its search crawler from a user-requested fetch. Google uses a different model: Google-Extended is a robots.txt product token, not a separate HTTP crawler identity.
These distinctions matter because allowing one agent does not automatically allow another, and blocking one does not necessarily control every way a provider can retrieve or use a page.
#1 Best Overall
- Durable Carbon Steel: Rack mount screws and cage nuts are made of high-quality carbon steel with a black finish for high strength and dependable durability.
- Easy Installation: Clear metric threads and uniform pitch for better grip. Nylon washers help secure screws and protect equipment surfaces.
- Organized Storage: All parts are packed in a portable storage box for easy organization and access.
- Wide Compatibility: Fits most square-hole racks and cabinets—ideal for server racks, network cabinets, equipment enclosures, and A/V gear.
- 20-Set Kit: Includes 20 mounting screws with nylon washers (M6 x 20 mm) and 20 square cage nuts—40 pieces in total—meeting daily install and replacement needs.
How do the major providers’ crawler controls differ?
| Provider and agent or token | Documented purpose | Relevant control and stated effect |
|---|---|---|
| OpenAI: OAI-SearchBot | Helps surface websites in ChatGPT search features. | OpenAI says its setting is independent of GPTBot. Blocking it means sites will not be shown in ChatGPT search answers, though they may still appear as navigational links. Search systems may take about 24 hours to adjust after a robots.txt update. OpenAI crawler documentation. |
| OpenAI: GPTBot | Crawls content that may be used in model training. | Its robots.txt setting is independent of OAI-SearchBot; an owner can allow search discovery while disallowing GPTBot. OpenAI crawler documentation. |
| OpenAI: ChatGPT-User | Supports certain user-triggered actions. | OpenAI says robots.txt rules may not apply to these requests. OpenAI crawler documentation. |
| Anthropic: ClaudeBot | Collects public-web material that could potentially contribute to training. | Anthropic says its bots honor robots.txt and supports Crawl-delay. It warns that blocking source IPs can interfere with reading robots.txt and may not provide a persistent opt-out; rules need to cover each relevant subdomain. Anthropic crawler documentation. |
| Anthropic: Claude-SearchBot | Improves search result quality. | Anthropic says its bots honor robots.txt. Anthropic crawler documentation. |
| Anthropic: Claude-User | Retrieves websites in response to individual user questions. | Anthropic says its bots honor robots.txt. Anthropic crawler documentation. |
| Google: Google-Extended | Controls whether content Google crawls may be used for specified Gemini model-training and grounding uses. | This is a robots.txt product token used with Google’s existing user agents, not a distinct HTTP crawler. Google says it does not affect Google Search inclusion or act as a Search ranking signal. Google crawler documentation. |
| Googlebot and page-preview directives | Googlebot manages crawling for Search, including Search AI features. | Google says Googlebot directives apply to Search AI features. Directives such as nosnippet, data-nosnippet, max-snippet, and noindex affect what is shown or indexed. Google AI-features guidance. |
| Perplexity: PerplexityBot | Automatically crawls to surface and link websites in Perplexity search results; Perplexity says it is not used to crawl content for foundation-model training. | Perplexity recommends allowing the crawler and its published IP ranges for search-result inclusion. Configuration changes may take up to 24 hours to reflect. Perplexity crawler documentation. |
| Perplexity: Perplexity-User | May fetch a page in response to a user question and include a link in its response. | Perplexity says this user-requested fetch generally ignores robots.txt. It recommends combining user-agent and IP checks in WAF rules and monitoring logs. Perplexity crawler documentation. |
Anthropic’s help documentation, dated April 7, 2026, says its bots respect “do not crawl” signals by honoring standard robots.txt directives. Provider labels, IP ranges, and instructions can change, so use each provider’s current documentation rather than treating this list as a permanent specification.
How do I block AI crawlers but allow search engine crawlers?
There is no universal “block AI, allow search” rule. Choose by provider and purpose. If you want a provider’s search product to find your pages but do not want to signal permission for model-development collection, use the provider’s distinct search and training controls where documented. OpenAI explicitly says OAI-SearchBot and GPTBot settings are independent; Google’s Google-Extended token affects specified Gemini uses rather than Google Search inclusion.
Rank #2
- Versatile Compatibility - The M6 rack mounting screw kit is designed for universal compatibility with most rack and cabinet systems with square holes. It is perfect for mounting 19 inch / 10 inch network cabinet, server cabinets, electronics enclosures, racks, shelf
- Length of M6 screws - The total length of the M6 screw is 19.7 mm (0.77 inches), the thread length - nominal length of the M6 screw is 16 mm (0.63 inches)
- Robust construction - These M6 screws and cage nuts are made of high-quality carbon steel and offer exceptional strength, corrosion resistance and durability, ensuring long-term performance even in extreme conditions
- Complete installation kit - Each pack contains 20 rack mounting screws, 20 square cage nuts and 20 washer plastic and provides a comprehensive solution for all your mounting needs and ensures you have enough material for different projects
- Effortless and efficient installation - With precise threads and a smooth design, these M6 screws allow easy insertion and secure attachment, optimise the installation process and improve work efficiency
Use the exact user-agent token or product token documented by the provider. A robots.txt group is only an example of policy syntax; it is not a complete site policy and must be checked against the rest of your file:
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This OpenAI example allows the named search crawler while disallowing GPTBot. It does not establish a rule for all OpenAI agents, all AI products, or user-triggered requests. Similarly, Google-Extended is not an HTTP user-agent group; follow Google’s instructions for that token in robots.txt. Avoid a broad site-wide disallow if your goal is to block only one agent.
Rank #3
- Package Contents: 50 High Quality Carbon Steel 50x M6 screws and cage nuts, 50x Plastic Washers
- Square cage nuts designed for Server Racks and Networking Cabinets with square holes, ideal for all Intellinet Rack Mountable equipment.
- Lifetime Mfg Warranty
- U.S TECHNICAL SUPPORT: We offer U.S technical Support
Blocking a search crawler can reduce a provider’s ability to surface your pages. Allowing it does not guarantee that a page will be cited, linked, ranked, or receive traffic. Blocking an agent used for user-triggered retrieval may prevent it from fetching your page for an individual request.
What robots.txt can and cannot do
Google describes robots.txt as a way to tell search crawlers which URLs they may access, used mainly to manage crawler traffic—not as a way to keep a page out of Google. A blocked URL can still be indexed based on links from other pages, even when its content is not crawled. For indexing or snippet controls, use the relevant page-level directives; for confidentiality, require authentication.
Rank #4
- Threaded hole hardware kit - 50 each #12-24 screws
- Fastens equipment to threaded hole rack mount rails
- Compatible with all #12-24 threaded hole racks
- Use robots.txt to express crawler-access preferences for agents that honor the file.
- Use noindex or preview controls when your goal is to limit indexing or what search results display. Apply the directive in a way the crawler can access and process; blocking the page in robots.txt can prevent it from seeing page-level instructions.
- Use authentication for private or sensitive information. Robots.txt is public and does not prevent a client from requesting a URL.
- Do not assume an opt-out reverses prior collection. A crawler rule governs future access behavior as implemented by the provider; it does not remove material a provider may already have collected.
Google’s guidance is explicit: “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.” Google Search Central: Introduction to robots.txt.
How to set and verify a crawler policy
- Choose the outcome for each purpose. Decide separately whether you want automatic search discovery, model-development collection, and user-initiated retrieval. Do not assume one provider’s controls map directly to another’s.
- Check each provider’s current instructions. Use its documented robots.txt token or user-agent and confirm what the rule controls. Review the whole robots.txt file for broader rules that could override or undermine your intended policy.
- Keep sensitive pages behind authentication. Use page-level indexing and preview controls when the goal is to change search visibility or snippets rather than crawler traffic.
- Review server and CDN access logs. Compare observed requests with the provider’s published IP information as well as its documented user-agent string. A user-agent can be imitated; it is not proof that a request came from the named provider.
- Check WAF and CDN rules separately. A page allowed in robots.txt can still be denied at the network layer. Perplexity specifically recommends combining user-agent and IP checks in WAF allow rules and monitoring logs.
- Allow for propagation, then recheck. OpenAI says search systems may take about 24 hours to adjust after a robots.txt update; Perplexity says configuration changes may take up to 24 hours to reflect. These are provider-specific estimates, not a universal guarantee. Check logs and the provider’s own documentation or webmaster tools after the change.
OpenAI and Perplexity publish IP information for identifying their crawlers; consult the current endpoints linked from their crawler documentation rather than copying an old range into a permanent rule. Anthropic cautions that blocking IPs can interfere with its ability to read robots.txt and may not persist as an opt-out. Prefer a documented purpose-specific directive where available, and apply Anthropic rules on every relevant subdomain.
Best Value
- Designed to support most servers fitting this rack rails depth 15-24" in a 19" standard rack or cabinet. Suitable for 4-post rack or 2 center weighted posts rack.
- Product Size: H 1.87" x W 2.70" x D 15"-24"(380-610 mm), Server holding board: W 1.97" x D 16.5", Depth adjustable from 15 inches to 24 inches, you can accommodate various rack sizes.
- Durable: Made of high quality cold rolled steel with powder coating, Support up to 110 lbs when installed to a 4-post rack or cabinet at 15 inches Deep.
- Including 1 Pair of 1U Server Shelf Rails, 8 sets of M6 screws and cage nuts.
- Easy Installation: Including all the screws for assembly and installation.
Does robots.txt stop AI training?
It can communicate a crawler preference to a provider whose documented agent honors robots.txt, but its effect depends on the provider, agent, and policy implementation. Google says Google-Extended controls specified Gemini model-training and grounding uses, while Perplexity says PerplexityBot is not used to crawl content for foundation-model training. OpenAI describes GPTBot as crawling content that may be used in training and separates its setting from OAI-SearchBot.
These controls do not secure a page, compel every client to comply, or establish what has already been collected. If the issue is access to confidential material, use authentication; if it is search indexing or snippets, use the relevant indexing or preview directives.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minute
