The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →To ask AI crawlers not to fetch your site, add crawler-specific Disallow rules to a robots.txt file at the root of each relevant website origin. To technically block requests, use server, firewall, WAF, or edge controls instead; robots.txt is a voluntary instruction, not access control. Choose the bot rules by purpose: blocking training crawlers is different from blocking search or user-directed retrieval, and the latter can reduce your site’s visibility in AI answers.
Decide what you want to block
AI companies may use separate crawlers for different functions. OpenAI identifies GPTBot with potential model-training collection and OAI-SearchBot with ChatGPT search. Anthropic lists ClaudeBot, Claude-User, and Claude-SearchBot for distinct crawling and retrieval purposes. OpenAI says each crawler setting is independent, so a rule for one does not automatically cover the others. See the operators’ OpenAI crawler documentation and Anthropic crawler guidance.
As an Amazon Associate I earn from qualifying purchases.
| Goal | Example user-agent token | Effect and tradeoff |
|---|---|---|
| Request exclusion from training collection | GPTBot or ClaudeBot |
Signals that the operator should exclude future content from the relevant training collection under its published policy. It does not necessarily block search or user-directed retrieval. |
| Block AI search crawling | OAI-SearchBot or Claude-SearchBot |
Can reduce or remove the site from that operator’s search answers. OpenAI says opted-out sites will not be shown in ChatGPT search answers, although navigational links may still appear; Anthropic warns that disabling search bots can reduce search visibility. |
| Block user-directed retrieval | Claude-User |
Prevents that bot from fetching content in response to a user’s request when the operator honors the rule, but can reduce the chance that the site is retrieved for those queries. |
| Block requests technically | Server, firewall, WAF, or edge rule | Can deny access rather than merely express a preference. A broad rule can also block legitimate users or useful crawlers if it is not scoped carefully. |
Do not treat those examples as a permanent, complete list. Check each operator’s current documentation for the bot name and purpose before adding a rule.
Free tools Windows power users keep installed
One-click scans. No signup required.
Add rules to robots.txt
Create or edit a plain-text file named robots.txt in the root of the site origin. For example, a file served at https://example.com/robots.txt can include these separate groups:
#1 Best Overall
- BEFORE YOU BUY: Our security mounting plates attach to the bottom rail of wood picture frames so a wall-mounted T-head screw can lock the frame in place. Designed for wood frames only. Not for metal frames, shelves, TVs, or structural loads.
- PACKAGE CONTENTS: Includes 100 security mounting plates and 200 #6 x 1/2 inch screws. Complete picture frame hardware and picture frame mounting hardware set for securing multiple frames, artwork, and wall displays.
- ANTI THEFT SECURITY: Designed as anti theft picture hangers and anti theft hangers, these plates lock into a wall-mounted T-screw to help prevent unauthorized removal. Also used as earthquake picture hangers to keep frames secure during vibration.
- DURABLE STEEL CONSTRUCTION: Zinc-plated steel frame mounting hardware and picture frame brackets resist bending and corrosion. Reliable picture frame fasteners and framing brackets designed for galleries, offices, schools, and public spaces.
- VERSATILE FRAME HARDWARE: Works with many hanging hardware for wall art systems including offset clips, picture clips, canvas clips, and z bracket installations. Ideal for securing framed art, photos, and wall displays.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /This requests that the named crawlers not crawl any path on that origin. Add only the groups for the uses you intend to exclude. To block search crawling as well, add its documented token, for example:
User-agent: OAI-SearchBot
Disallow: /Google documents user-agent, allow, and disallow as supported robots.txt fields. For path-specific limits, replace / with the path you want the crawler to avoid, and verify the result against the relevant operator’s rules. See Google’s robots.txt introduction.
Rank #2
- FIDO2 & Passkey Ready: Business-ready and FIDO2 L1 certified. This key is supported by major management suites and is ideal for both individual and enterprise deployment. Works seamlessly with Gmail, Facebook, GitHub, Dropbox, Coinbase, and more.
- Universal Connectivity (USB-A ): Features a built-in USB-A connector—simply unfold the key and plug it into your compatible PC or laptop for seamless authentication on the go.
- Dedicated Manager App: Use the Thetis Manager App for the initial hardware PIN setup. Setting the PIN on the device first ensures a smooth registration process. Once the PIN is configured, you can begin registering the key across your favorite FIDO2-compatible online services.
- Ultra-Durable & Portable: Featuring a rotating metal cover, this key is water, crush, and tamper-resistant. It fits easily on a keychain and requires no batteries or network connectivity.
- Check FIDO2 compatibility before purchase - Known limitations: ID Austria is not supported (requires FIDO2 Level 2). Windows Hello login only works with Windows Enterprise editions that support Entra ID, and NFC is NOT supported.
Apply the rules to every origin you mean to cover
A robots.txt file applies only to the protocol, host, and port from which it is served. A file on https://example.com does not govern https://shop.example.com, another protocol, or another port. Publish the rules on each relevant origin and subdomain; Anthropic likewise advises a robots.txt file for each subdomain you want to opt out. Google explains this scope in its robots.txt documentation.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchUse deliberate groups and syntax
Give each crawler or set of crawlers that should receive the same rule an appropriate user-agent group. Google’s parser selects the most specific matching user-agent group rather than combining it with the wildcard group. A disallow rule prevents crawling under the applicable rules, but does not guarantee that a URL will be removed from search results. For non-standard directives such as Crawl-delay, confirm support with the specific operator: Anthropic says it supports the extension, while Google says its parser does not.
Rank #3
- Ultra-Compact FIDO2 Security Key - Plug-and-stay or carry on a keychain. This USB-A hardware security key offers portable, always-on protection for desktop and mobile use. (Item Size: 0.75 X 0.74 IN x 0.25 IN)
- USB-A Hardware Key for All Devices - Works with USB-A ports on PC, Mac, Android, and other laptop/notebook device. Enables secure, cross-platform login with FIDO2.0 passkey support.
- FIDO Certified Security Key - Meets FIDO and FIDO2 standards. Works with Google, Microsoft, GitHub, Dropbox, and more. Please check service compatibility before purchase.
- Passwordless Login with Passkey - Supports passkey login via WebAuthn and CTAP2. Enjoy password-free sign-ins where supported. Not all websites or services currently support passkeys.
- Advanced Multi-Factor Authentication - Offers 200 FIDO2 passkey slots and 50 OATH-TOTP slots. Strong, flexible 2FA/MFA support across various apps and authentication platforms.
Check what the site actually serves
- List the origins and purposes. Identify every host or subdomain to cover, then decide separately whether to exclude training collection, AI search, or user-directed retrieval.
- Publish the matching groups. Use the operator’s documented user-agent token and the appropriate path in each group.
- Fetch each live file. Open
/robots.txton every origin and confirm the server returns the intended text. Editing a file in a hosting dashboard is not enough if a CDN or another server layer serves a different version. - Review enforcement rules separately. Check server, CDN, firewall, and WAF settings for conflicting policies or overrides. Cloudflare notes that managed robots.txt can be combined with an existing file in its managed robots.txt documentation.
- Recheck after changes. Bot names, products, and controls evolve, so verify the live file and edge configuration after deployment and when an operator updates its documentation.
When robots.txt is not enough
Robots.txt communicates a crawl preference to compliant bots; it does not prevent a request or stop a crawler that ignores the instruction. Cloudflare explicitly describes compliance as voluntary. If you need an enforced block, deny the traffic at the server, firewall, WAF, or edge. Cloudflare’s AI bots documentation describes AI Crawl Control and policy options for classes such as Search, Agent, and Training, including whole-zone blocking, blocking on ad-displaying pages, or allowing classified traffic. Current dashboard choices and defaults may change, so check their effect before rollout. Some bots can belong to more than one behavior class.
Cloudflare’s bot reference includes tokens such as GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-CloudVertexBot, Bytespider, CCBot, meta-externalagent, meta-externalfetcher, Applebot, Amazonbot, and DuckAssistBot. Use this as a starting point, not a universal blocklist: confirm each token’s purpose with its operator. Cloudflare says Bot Management customers can use detection IDs in WAF rules; other plans can use user-agent matching in robots.txt or WAF rules. Its verified bots reference points to a live directory, which is preferable to assuming a static list stays current.
Quick Recap
Best Value
- 【Anti-Theft Post Attachment Kit】 Effortlessly & Securely Fastens Signs, Compatible with 3/8" Holes in U-Shaped Channel Posts, Square Metal Posts & Tubular Posts
- 【Anti-Theft Design】 Featuring an anti-theft beveled-edge nut and one-way security bolt, our post attachment kit effectively prevents removal with ordinary tools
- 【Excellent Quality】Made of high-quality superior metal and finished with zinc coating, Fengone sign attachment kit stays rust-free in damp or wet environments.
- 【Installation】1. Hand-tighten the first nut onto the signpost’s back 2. Tighten the second nut upside-down on top of the first—they lock together. 3. Insert a wrench between the two nuts and tighten to secure 4. Post-tightening, remove the 2nd nut and save for future removal or reinstallation
- 【Package Inculde】8 PCS 2.5" Bolts, 12 PCS Anti-Theft Nuts. If you have any questions about our products, please feel free to contact us, and we will give you a satisfactory solution
Rank #4
- 🔒 Product Dimensions: You get a fresh set of 8 top-notch security screws, each measuring 1.7 x 0.38 x 0.38 inches, expertly made by Honkador!
- 💪 STRONG & DURABLE: These door hinge screws feature a unique, hardened design that boosts your door's security. They're specially crafted to fit snugly in hinge leaves and door frames, safeguarding your home from intruders and making it harder for burglars to break in.
- 🛠️ EFFORTLESS INSTALLATION: Swapping out your old hinge screws for these security ones is a breeze. Just take out one hinge screw and its match from the door, then slide the new security screw into place for a hassle-free setup.
- 🚪 WIDE APPLICATIONS: Perfect for entry doors, residential and commercial spaces, as well as interior doors, offices, computer rooms, and archives. These locking screws work wonders with door hinges, black hinges, wood screen doors, and interior shutters. Remember to check your door hinges for the right fit!
- 💬 AFTER-SALES SUPPORT: Got any questions or issues with your product? Just shoot us an email, and we promise to get back to you within 24 hours to make sure you're happy and help you out with any concerns!
Choose the narrowest rule that meets your goal
- If your concern is model-training collection, block the documented training crawler without automatically blocking that company’s search or user bots.
- If you also want the site excluded from an AI search experience, block its search crawler and account for the resulting visibility loss.
- If you need to prevent requests rather than signal a preference, use an enforcement control and test that it does not catch legitimate site visitors or search services you intend to keep.
- If you are unsure of a token’s function or impact, confirm it in the operator’s current documentation before deploying the rule.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
