Recommended Free Tools
robots.txt, noindex and AI crawler rules control different things: what a crawler may fetch, whether a supported search engine may list a page, and how a particular provider may use crawled content. Choose the directive for the outcome you want—and keep a page accessible whenever a crawler needs to read an indexing or preview instruction.
What each control does
| Control | What it governs | How it is applied | Main limitation |
|---|---|---|---|
robots.txt |
Whether compliant crawlers may fetch URL paths. | A text file at the site’s top level, with rules for crawler user-agent groups. Its scope is the host, protocol and port where it is served. | It does not reliably remove URLs from search results or secure content. A crawler blocked from a page cannot read page-level directives there. |
noindex |
Whether a supported search engine should include a page or resource in its results. | An HTML robots meta tag, or an HTTP X-Robots-Tag response header. The header can cover non-HTML files such as PDFs. |
The crawler must be able to fetch the resource and process the directive. Updating results can take time. |
| AI crawler controls | A named provider’s crawler and specified uses of content. | Usually provider-specific user-agent rules in robots.txt, such as Google-Extended, OAI-SearchBot or GPTBot. |
There is no universal AI opt-out. Tokens and purposes differ by provider, and robots rules are not access security. |
| Search preview controls | How much page content appears in supported search features. | Google documents nosnippet, data-nosnippet, max-snippet and noindex for controlling information shown in its Search AI features. |
These affect Google Search presentation; they are distinct from Google-Extended’s specified controls for other Google systems. |
Does robots.txt remove a page from Google?
No. A robots.txt block tells compliant crawlers not to fetch matching paths; it is not an instruction to remove those URLs from results. Google says a blocked URL may still be indexed if it is discovered through links from other pages. The page may appear without content Google could not crawl. See Google’s Robots.txt Introduction and Guide and Control the Content You Share on Search.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
HYBRID ALGORITHM FOR ENHANCING FOCUSED WEB CRAWLING USING BLOCK SEGMENTATION | $2.76 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
When to use noindex
Use noindex when the goal is to keep a page or resource out of supported search results. The directive must be visible to the search crawler, so do not block the URL in robots.txt at the same time you expect Google to read its noindex.
Free tools Windows power users keep installed
One-click scans. No signup required.
HTML pages
Add a robots meta tag in the page’s <head>, for example <meta name="robots" content="noindex">. Google must be able to fetch the page to process it.
PDFs and other non-HTML resources
Send an X-Robots-Tag: noindex HTTP response header for the resource. Google does not support noindex rules in robots.txt. Details are in Google’s Block Search Indexing with noindex.
Google-Extended is not a Google Search opt-out
Google-Extended is a standalone robots.txt token for controlling whether content accessed by Google crawlers may be used to train future Gemini models and for grounding in specified Gemini products. Google says it does not affect inclusion in Google Search or act as a Search ranking signal. The token is used in robots.txt; it does not have its own separate HTTP request user-agent string.
For Google AI features that appear within Search, Google says Googlebot directives govern access. Its documented controls for limiting information shown in those features include nosnippet, data-nosnippet, max-snippet and noindex. Google’s common crawlers documentation and AI Features and Your Website explain the distinction.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →OpenAI separates ChatGPT search from potential training use
OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT search features, and GPTBot as associated with potential use of crawled content in training generative AI foundation models. OpenAI says the settings are independent: a publisher can allow one and disallow the other. Check its Overview of OpenAI Crawlers for the current tokens and guidance.
OpenAI’s publisher FAQ also says that, in some circumstances, ChatGPT Atlas may surface a disallowed page’s link and title if it discovers the URL through another search provider or by crawling other pages. The FAQ says a publisher can use a noindex meta tag to prevent that, but the crawler must be allowed to fetch the page to read it. This is OpenAI-specific guidance, not a rule that should be generalized to every AI product. See Publishers and Developers – FAQ.
Choose a control by the result you want
- Reduce fetching by a compliant crawler: Add a robots.txt rule for the relevant provider token and paths. This communicates a crawler preference; it does not protect confidential information.
- Keep a page out of Google results: Leave it crawlable and serve
noindexin the HTML or response header. - Limit Google Search snippets or AI Search content: Use the Search preview and indexing controls Google documents, rather than treating
Google-Extendedas a Search control. - Allow ChatGPT search discovery but disallow potential training use: Configure
OAI-SearchBotandGPTBotseparately according to OpenAI’s current documentation. - Keep content private: Require authentication or remove the content. Do not rely on robots.txt, which is public and does not stop noncompliant access.
Implementation details that can trip you up
Google fetches robots.txt before crawling and uses it to determine which paths it may crawl. Google’s robots.txt documentation states a 500 KiB processing limit; content after that point is ignored. The file applies only to the host, protocol and port where it is served, so a rule on one host or protocol should not be assumed to govern another. See How Google Interprets the robots.txt Specification.
Google Search Central summarizes the purpose this way: “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.” That is a crawl-access instruction, not a promise of deindexing or confidentiality.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
