Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →You can use robots.txt to ask a named, compliant AI crawler not to fetch pages on a particular host. To block a crawler across the whole site, add a user-agent group for its documented token and set Disallow: /. This is a crawl preference, not a privacy or access-control measure: crawlers may ignore it, and blocked URLs may still appear in search results.
Choose which AI crawlers to block
AI services may use separate crawlers for model training, search, and pages fetched at a user’s request. Decide which activity you want to restrict before adding rules: blocking one token does not necessarily block the provider’s other crawlers, and may affect how its service finds or retrieves your pages.
As an Amazon Associate I earn from qualifying purchases.
| Provider and token | Documented purpose | What blocking it may affect |
|---|---|---|
OpenAI: GPTBot |
May collect content used to train generative AI foundation models. | Access for that training-related crawler. OpenAI documents this setting separately from its search crawler. |
OpenAI: OAI-SearchBot |
Used to surface websites in ChatGPT search features. | Whether sites can be surfaced through that search crawler. Its setting is independent of GPTBot. |
OpenAI: ChatGPT-User |
Fetches pages in response to user actions. | robots.txt rules may not apply to these user-triggered visits. |
Anthropic: ClaudeBot |
May collect content that could contribute to model training. | Access for this crawler, rather than every Anthropic access path. |
Anthropic: Claude-SearchBot |
Used to improve search result quality. | Search-related access and potentially search visibility. |
Anthropic: Claude-User |
Retrieves pages at a user’s direction. | User-directed retrieval. |
These purposes and distinctions are described in the providers’ documentation: OpenAI’s crawler overview and Anthropic’s crawler guidance. Use the exact token the operator documents; do not assume a provider has one catch-all crawler. Google’s crawler directory lists Google-Extended, but the specific effects of disallowing it are not established here, so do not infer them from the token’s name.
Add a crawler-specific rule to robots.txt
For a site-wide request to a named crawler, the basic pattern is:
User-agent: GPTBot
Disallow: /Replace GPTBot with the exact crawler token you intend to address. Repeat a separate group for each crawler you want to block. Anthropic’s documented example is:
User-agent: ClaudeBot
Disallow: /To restrict only a section rather than the whole host, replace / with the path you want the crawler not to fetch, and check the crawler operator’s parsing guidance. The Robots Exclusion Protocol defines user-agent product tokens and Allow/Disallow rules; crawler-specific parsing can differ. Google, for example, selects the most specific matching user-agent group for its crawlers.
Put the file at the root of every host you want covered
Google’s documentation specifies a UTF-8 text file named robots.txt at the root of the applicable site host. A rule applies only to the protocol, host, and port where that file is served. A file at https://example.com/robots.txt does not automatically govern https://www.example.com/, another subdomain, a different port, or the HTTP version of the host. Publish the appropriate file for each scope you intend to cover. Anthropic likewise advises placing its file in the top-level directory and repeating the opt-out for each subdomain where it is wanted.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11See Google’s guidance on robots.txt basics and limitations, creating and submitting a file, and how Google interprets the specification.
Verify the published rules and their scope
- Open the file directly. Visit
https://your-host/robots.txtand confirm that the published text contains the intended user-agent groups and directives. - Check every applicable host and protocol. Test each hostname and HTTP or HTTPS version you mean to cover; a file on one does not extend to the others.
- Test the syntax and matching behavior. Check the rules against the crawler’s documented parser guidance. Google provides a robots.txt testing tool for its crawlers.
- Review other access layers. A CDN, firewall, authentication layer, or server configuration can produce behavior different from the file. If you cannot access the site root to publish it, Google notes that you may need help from your hosting provider.
Know what robots.txt cannot prevent
It does not secure private content
The IETF’s Robots Exclusion Protocol states: “These rules are not a form of access authorization.” Google puts the practical limitation plainly: “The instructions in robots.txt files cannot enforce crawler behavior to your site; it’s up to the crawler to obey them.” The file is a request to crawlers, not authentication. A crawler that does not support or obey the rules may still request a disallowed page.
For content that must remain private, use server-side access controls such as authentication or password protection. Do not publish it openly and rely on a robots rule to keep it secret.
It does not guarantee removal from search results
A search engine may discover a disallowed URL through links to it and show the URL in results even if it cannot crawl the page body. The result may reveal the URL and public information such as anchor text. A crawl disallowance therefore is not a reliable way to hide a page from search.
Recommended Free Tools
For search visibility, use indexing controls or an appropriate removal process. An on-page noindex directive only works when the crawler can access the page to read it; blocking the page in robots.txt can prevent it from seeing that directive. Google explains this distinction in its robots.txt guide.
Quick Recap
Match the control to your goal
- Reduce requests from compliant AI crawlers: publish crawler-specific
Disallowrules. - Keep material private: require authentication or use server-side access restrictions.
- Control search indexing: use indexing controls or removal processes appropriate to the goal, rather than assuming crawl blocking removes a URL.
- Preserve selected AI access: block only the documented crawler tokens you want to restrict, and leave other groups unchanged unless you intend to affect them too.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
