To ask AI providers not to use your public website content for training, add each provider’s documented crawler token to your site’s root robots.txt and disallow the paths you want excluded. There is no single rule that covers every AI operator, and training crawlers may be separate from bots used for search discovery or user-requested retrieval. Choose the policy for each service deliberately: blocking a search crawler can reduce visibility in that provider’s search product.
What to put in robots.txt
For a site-wide request to the crawlers shown below, a starting example is:
As an Amazon Associate I earn from qualifying purchases.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /This is illustrative, not a universal opt-out. It disallows the OpenAI and Anthropic crawler tokens shown, and Google-Extended, while allowing OAI-SearchBot. Before publishing, confirm each token, its documented purpose, supported directives, and the paths you intend to govern against the operator’s current guidance. An allow rule does not make a crawler fetch your pages; hosting, CDN, firewall, or bot-management rules can still deny requests.
Choose training and discovery settings separately
For each operator, identify whether a crawler is used for training, search indexing or discovery, or fetching content in response to a user request. Those purposes can have different controls and consequences.
#1 Best Overall
- No more exposed information in unprotected notary journals. This product shields clients' confidential information from prying eyes. It allows the Notary Public to keep the journal open during the transaction, as NO prior client information is viewable.
- Shields clients' AND Notaries Public' confidential information
- GLBA and HIPAA require strict confidentiality policies and procedures. Notary Privacy Guard is a compliance tool for the professional Notary Public.
- Decreases Notary Public's liability from exposing client information
- Journal column headers are printed on the Notary Privacy Guard, no having to peek underneath to complete the journal entry. Becomes part of the journal and also acts as a place marker.
| Operator | Documented control and purpose | Visibility consideration |
|---|---|---|
| OpenAI | GPTBot relates to potential training use; OAI-SearchBot supports surfacing websites in ChatGPT search. See OpenAI’s crawler overview. |
OpenAI describes the settings as independent. Disallowing OAI-SearchBot can affect visibility in ChatGPT search. Updated robots.txt may take time to be reflected in search systems. |
Google-Extended is a standalone robots.txt token for whether content Google crawls may be used for certain Gemini model training and grounding. See Google’s crawler documentation. |
Google says Google-Extended does not affect inclusion in Google Search or act as a Search ranking signal. Search visibility and Search AI feature controls use Googlebot and other Search controls instead. | |
| Anthropic | ClaudeBot is documented as a potential training crawler; Anthropic provides a robots.txt method to block it. See Anthropic’s crawler guidance. |
The cited guidance establishes the crawler preference, not a separate search-product visibility effect. Check Anthropic’s current documentation for scope and crawler identities. |
Cloudflare’s bot reference distinguishes AI crawlers, AI search, and assistants, illustrating why a generic AI-bot blocklist can miss meaningful differences.
How to add or update your site’s policy
- Decide what you want to block. Make a separate choice for each operator and purpose. If you want to keep your content discoverable in a search product, do not disallow that product’s search crawler without accepting the potential visibility effect.
- Edit the root robots.txt file. Add a group for each exact user-agent token in the operator’s current documentation. For example, the file is typically available at
https://your-domain.example/robots.txt. Use the actual hostname and file path for your site. - Set the intended path rules.
Disallow: /requests that the matching crawler avoid the whole site; a narrower path can limit the request to that path. Do not assumeUser-agent: *makes the same purpose-specific choice as a named token. - Check other access layers. Review your host, CDN, firewall, and bot-management settings. A robots.txt allow rule does not override a network block. Cloudflare documents tools for managing crawler policies and bot categories in its bot documentation.
- Verify the live file and monitor requests. Fetch the public robots.txt URL to confirm the saved content is served, then review server or CDN logs for requests from the relevant user agents. A robots.txt entry is a crawler request, not proof that a bot has obeyed it.
What robots.txt can—and cannot—do
Robots.txt expresses a preference to crawlers that choose to honor it; it is not authentication or a security barrier. Google notes that some crawlers may not obey robots.txt, and that the file is not the way to keep sensitive pages out of Search. Keep private material behind authentication or remove it from public access. For Google Search removal or control of snippets and previews, use the appropriate Search controls described in Google’s AI features documentation, including controls such as noindex, nosnippet, data-nosnippet, and max-snippet where applicable.
Rank #2
- No more exposed information in unprotected notary journals. This product shields clients' confidential information from prying eyes. It allows the Notary Public to keep the journal open during the transaction, as NO prior client information is viewable.
- Shields clients' AND Notary Publics' confidential information
- GLBA and HIPAA require non-disclosure policies and procedures. Notary Privacy Guard is a compliance tool for the professional Notary Public.
- Decreases Notary Public's liability from exposing client information
- Journal column headers are printed on the Notary Privacy Guard, no having to peek underneath to complete the journal entry. Becomes part of the journal and also acts as a place marker.
A crawler disallow rule addresses future crawling requests within the rule’s scope; the sources cited here do not establish that it deletes material already collected or removes content from an existing model. Do not treat a robots.txt change as a retroactive deletion request.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Keep the policy current
Provider tokens, bot purposes, and product behavior can change. Recheck each operator’s official documentation when you update the file, and compare it with your server or CDN logs. Cloudflare’s categories are also useful context, but its list is not a substitute for the operator’s stated token and policy.
Quick Recap
Best Value
- 8 ¾ x 11 inches, spiral bound soft cover.
- Block style entry, 200 entries per journal
- Privacy guard protects client information
- For use in any state
- Electronic & remote notarization option
Rank #4
- Value Pack: Our password keeper refill comes with 216 pages 80gsm paper and 12 durable laminated dividers with alphabetical tabs. for password organizer section each page has 3 entries, total allows 576 records of website, username/ID, password/hint, name, phone, email, security questions/notes etc., 12 pages/72 records of Software. license number and purchase date etc., 6 lined pages for important things to remember, 1 page for emergency information and 1 PVC protect film.
- Premium Quality: 80gsm off-white paper which will protect your eyes from strong lights and viewing strain and allows smooth writing and reducing ink leakage, erase fraying and shade issue. 12 film laminating durable dividers with alphabetical tabs for easy scrolling of your search.1pc PVC sheet protects all inner pages from wetting.
- Fits A5 6 Ring binder: Fits binder cover No smaller than 6.7" W x 9.25" H x1" Thick. Divider is 5.6” W x 8.19” H, inner page is 5.2" W x 8.2" H, 212 pages/106 sheets, both sides printing, 6 holes punched (hole space is 0.75in/19mm, hole space between 3rd & 4th holes is 2.76in/70mm, dia 0.197in/5mm). Suggest match this large print passwords book refills with A5 lockable binder whose size is large than 9.25"x6.7"x1" for perfect combination.
- Pairs perfectly with our hardcover refillable password book with lock B09FGZ1CDF. Keep your important internet passwords, website, username/ID, password/hint, name, phone, email, security questions/notes. license and purchasing date with this pack of refill pages for perfect internet password keeper,huge space to store all your passwords and account & website login details in one place ,fully protect your personal privacy and keep online website account information & user data safe.
- 100% money back if you're not satisfied with our products. Any questions, don't hesitate, just contact us!
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
