Blocking GPTBot does not, by itself, remove your site from ChatGPT Search. OpenAI treats GPTBot and OAI-SearchBot as separate controls: the former relates to content that may be used to train OpenAI’s models, while the latter is used to surface websites in ChatGPT Search. To preserve Search eligibility while signaling that GPTBot-crawled content should not be used for training, disallow GPTBot and leave OAI-SearchBot allowed.
What each OpenAI crawler does
OpenAI’s Overview of OpenAI Crawlers describes these user agents as serving different purposes. They are not interchangeable, and changing one robots.txt group does not automatically change the others.
As an Amazon Associate I earn from qualifying purchases.
| User agent | Purpose | What its robots.txt setting controls |
|---|---|---|
GPTBot |
Crawls content that may be used to train OpenAI’s generative AI foundation models. | Signals whether GPTBot may crawl the site; disallowing it is the relevant robots.txt choice for opting out of potential training use of content it crawls. |
OAI-SearchBot |
Used to surface websites in ChatGPT Search. | Controls crawler access relevant to Search visibility, including summaries and snippets. OpenAI says not to block it if you want site content included in those results. |
ChatGPT-User |
Used for certain user-initiated visits to pages. | It does not determine Search inclusion. OpenAI says robots.txt rules may not apply to these user-initiated actions. |
OpenAI says the settings are independent: a webmaster can allow OAI-SearchBot for Search results while disallowing GPTBot to indicate that its crawled content should not be used for training. This is a signal about potential use of content GPTBot crawls, not a guarantee that material cannot reach OpenAI by any other route.
Recommended Free Tools
Choose the robots.txt rules for your goal
Keep ChatGPT Search eligibility while disallowing GPTBot
Use separate groups for the two crawlers. For example:
#1 Best Overall
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /This signals that GPTBot should not crawl the site while OAI-SearchBot may crawl it for ChatGPT Search. OpenAI’s publisher guidance says to make sure OAI-SearchBot is not blocked for content to be included in ChatGPT summaries and snippets. The configuration makes pages eligible for crawling; it does not guarantee that a particular page will appear in a Search answer.
Block ChatGPT Search crawling
If you do not want OAI-SearchBot to crawl your site for Search, use a separate group that disallows it:
User-agent: OAI-SearchBot
Disallow: /This is the relevant robots.txt control for ChatGPT Search crawling, not the GPTBot rule. OpenAI says opting out of OAI-SearchBot means a site will not be shown in ChatGPT Search answers, though it may still appear as a navigational link.
Free tools Windows power users keep installed
One-click scans. No signup required.
Set only the policy you intend
Review your existing file before adding rules. A broad rule such as User-agent: * can affect many crawlers; crawler-specific groups let you express different choices for GPTBot and OAI-SearchBot. Robots.txt rules are public signals about crawling, not access controls for private information. Protect confidential pages with authentication or another access-control method.
Why a blocked page can still leave a trace
Disallowing a page in robots.txt does not necessarily prevent every display of its URL. OpenAI’s Publishers and Developers FAQ says a page’s link and title may still appear in ChatGPT Atlas if OpenAI learns the URL from a third-party search provider or another crawled page and has signals that the page is relevant.
For the specific link-and-title outcome described in that FAQ, OpenAI recommends a noindex meta tag. The page must be crawlable so the crawler can read the tag; blocking the crawler in robots.txt can prevent that. Add the tag to the page’s HTML head:
<meta name="robots" content="noindex">A noindex tag is distinct from a robots.txt disallow rule: the crawler must fetch the page to see the tag. If your goal is to keep a page private, use authentication rather than relying on either directive.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCheck the whole access path, not just robots.txt
A crawler may be allowed by robots.txt and still be unable to fetch a page. Firewalls, CDNs, bot-mitigation systems, verification challenges, authentication requirements, geographic restrictions, and rate limits can all affect access. OpenAI discusses these access-layer issues in its Advertiser Guidance for Allowing OpenAI Web Crawlers; that guidance is written for ad landing-page validation, so its ad-specific instructions should not be treated as controls for ChatGPT Search.
Quick Recap
- Check that the crawler-specific group in robots.txt expresses the policy you want.
- Check whether a CDN, firewall, bot challenge, or rate limit blocks the relevant crawler despite the robots.txt rule.
- Check authentication and geography restrictions if the affected page is not generally accessible.
- After changing robots.txt, allow about 24 hours for OpenAI’s search systems to adjust, as OpenAI states this is an approximate processing window.
Which setting answers each question?
- “Does blocking GPTBot remove my site from ChatGPT?” No. It does not, by itself, block ChatGPT Search; OAI-SearchBot is the Search-related crawler.
- “Which robots.txt user-agent should I block to keep pages out of ChatGPT Search?” OAI-SearchBot. OpenAI says blocking it prevents a site from being shown in Search answers, though a navigational link may still appear.
- “Can I allow Search while disallowing GPTBot?” Yes. OpenAI describes those settings as independent; use separate user-agent groups to make the choices clear.
- “Will robots.txt prevent a URL and title from appearing in ChatGPT Atlas?” Not necessarily. For the described Atlas display, OpenAI recommends noindex, and the crawler must be able to read the page to see that tag.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
