Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Blocking AI crawlers is not a single yes-or-no decision. It depends on which crawler you mean, what that crawler is used for, and whether you want the product that uses it to show your pages. Some operators publish separate controls for AI search and for model training, so a site can allow one and refuse the other. robots.txt is the mechanism for those controls, and it works only for crawlers that choose to honor it. This guide explains the documented OpenAI and Google tokens, what each block costs you, sample rules you can adapt, and where robots.txt stops being useful.
What robots.txt can and cannot do
robots.txt is a plain text file at the root of a site that asks crawlers not to fetch certain paths. The IETF standard for it, RFC 9309, describes the rules as requested crawler behavior and states: “These rules are not a form of access authorization.” In other words, a well-behaved crawler reads the file and skips what you disallowed; a crawler that ignores the file is not stopped by it. Treat robots.txt as a stated preference, not a lock.
As an Amazon Associate I earn from qualifying purchases.
Four separate decisions
Most confusion comes from treating “AI crawlers” as one category. Make these four decisions separately:
Free tools Windows power users keep installed
One-click scans. No signup required.
- Search and referral visibility: Do you want an AI search product to discover and link to your pages?
- Training and other reuse: Do you want your content used to train models, or used for grounding answers in certain products?
- Server load and crawl behavior: Is a particular bot fetching so often that it strains your server?
- Confidentiality: Must the content stay private? If so, robots.txt is the wrong tool (see below).
The crawler tokens with documented controls
The table below covers the tokens that OpenAI and Google document at the time of writing. The list is not exhaustive, and other operators use their own names.
| Token | Operator | Documented purpose | Effect of disallowing it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search features | Opted-out sites are not shown in ChatGPT search answers, though they may still appear as navigational links |
| GPTBot | OpenAI | May crawl content to train OpenAI’s generative AI foundation models | OpenAI says this setting is independent of OAI-SearchBot |
| ChatGPT-User | OpenAI | Fetches triggered by an individual user; not the automatic search crawler | Not a reliable control: OpenAI cautions that robots.txt rules may not apply to user-initiated visits |
| OAI-AdsBot | OpenAI | Concerns pages submitted as ChatGPT ads | Not stated in the sources checked; confirm in OpenAI’s crawler documentation |
| Googlebot | Crawls pages for Google Search | Affects crawling for Google Search | |
| Google-Extended | robots.txt control token for crawled content used in future Gemini model training and grounding | Does not affect Google Search inclusion or rankings |
OpenAI’s crawlers
OpenAI documents OAI-SearchBot and GPTBot as separate controls. OAI-SearchBot is the one that matters for ChatGPT search visibility, and GPTBot is the one tied to training. Because they are independent, you can allow the first and disallow the second. OpenAI also says its ChatGPT search systems may take about 24 hours to adjust after a robots.txt change, so don’t judge a new rule by what happens in the first day.
Google’s tokens
Google-Extended is not a crawler you will find in your logs. It has no separate HTTP user-agent string and works only as a robots.txt control token. Googlebot still handles Search. Disallowing Google-Extended governs use of your content for Gemini training and grounding, and Google states that it does not change Search inclusion or ranking.
Blocking a URL is not removing it from search
Disallowing a path stops cooperating crawlers from fetching it, but Google notes that a URL can still appear in results if other pages link to it. If your goal is to keep a page out of search results, you need a noindex directive, and the crawler must be able to fetch the page to see it. A page disallowed in robots.txt never reaches that directive. Use robots.txt for crawl control and noindex for removal from search; use real access controls for anything confidential.
Recommended Free Tools
Three approaches
Keep AI search visibility, limit training use
Allow the search-facing token and disallow the training-facing token where the operator documents them separately. For OpenAI, that means allowing OAI-SearchBot and disallowing GPTBot. For Google, disallowing Google-Extended limits Gemini training and grounding while leaving Googlebot’s Search crawling untouched. This is the approach most sites with a visibility goal should start from.
Rank #3
- 【Tired of constantly searching for or resetting your passwords?】 MOSA BEAR password keeper book is the perfect solution for you! This password book provides a dedicated place to securely store all your important website addresses, emails, usernames and passwords, ensuring your information is protected and easy to find. The well-designed log pages help you manage multiple accounts in a systematic way, saying goodbye to password confusion.
- 【Premium Design & Password Security】 The password book with alphabetical tabs features an anonymous cover design with no title on the cover, effectively avoiding information exposure. The password keeper design is specifically designed with password security in mind, providing space to record password hints instead of writing directly on the password itself, further protecting your important information.
- 【Simple Layout and Plenty of Space】The 160-page password logbook is designed to provide ample space to record passwords and other important information. It can store up to 414 passwords. In addition, it provides extra pages to record other information, such as email setup, card information, computer operating system information, software licenses, and more. The journal also includes 3 blank pages at the end for you to add additional notes.
- 【Palm-sized Size & Premium Quality】 This password notebook has an ideal size, 4.3" x 5.7", for carrying around, whether in a purse or pocket. Its sturdy glue binding allows the notebook to unfold smoothly and is more comfortable to use. The inner pages are made of high-quality 100GSM thick paper, which can effectively reduce ink penetration and ensure a cleaner and neater writing effect. The overall design takes into account both portability and durability, making it an ideal choice for recording important passwords.
- 【A-Z Tabs for Quick Search 】Our password book comes with alphabetical tabs to help you find the password you need quickly and easily. Alphabetically organized tabs ensure that you can quickly flip to the right section, saving you the time and hassle of searching for your password.
Block a provider broadly
Disallow all of a provider’s documented tokens. Expect to lose visibility in that provider’s search or answer product if its search token is among those you block. Check the provider’s current documentation before you deploy, because token names and purposes change.
Protect confidential material
Require authentication, or otherwise restrict access on the server. Google advises password protection or other access controls for private content, and RFC 9309 makes the same point about robots.txt itself. Do not rely on a disallow rule to keep anything secret.
Rank #4
- Bookbound planner helps you keep track of passwords and favorite websites
- Room for over 200 entries; 3.5 x 6 inch page sizes
- User name and security questions field
- Tips for what makes a strong password; web resources; notes pages
- Printed on quality paper containing 30% post-consumer waste; black simulated leather cover; 3.63 x 6.13 x .21 inches
Sample policies
The first sample keeps ChatGPT search visibility and opts out of OpenAI training and Google’s Gemini training and grounding. Googlebot has no rule here, so it keeps normal access.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Google-Extended
Disallow: /
The second sample removes both OpenAI tokens. This will reduce visibility in ChatGPT search, and it does not address ChatGPT-User fetches, which OpenAI says robots.txt may not govern.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
Avoid a universal rule like User-agent: * followed by Disallow: / if your intent is to block AI bots only. That rule tells every cooperative crawler, including ordinary search engines, not to crawl the site at all.
Implementation checks
- Place the file at the top level of the site. Google applies it to the same protocol, host, and port, so
example.com,www.example.com, and any subdomain that serves content each need their own file. - Paths are case-sensitive in Google’s implementation, so
/Private/and/private/are different rules. - robots.txt is publicly readable. Do not list sensitive directory names in it.
- Allow about 24 hours for OpenAI’s ChatGPT search systems to pick up a change.
- Google-Extended will not show up as its own bot in server logs, so verify its effect by reading the file, not by searching logs.
Scope of this guide
The controls described here come from OpenAI’s crawler documentation, Google’s crawler and robots.txt documentation, and RFC 9309. This guide does not cover crawler tokens or purposes for Anthropic or other AI operators. For those, use each operator’s own current crawler documentation, and confirm the token names before writing rules.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
