Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
If an AI search service cannot find your pages, inspect the live robots.txt file before changing your site or submitting URLs. Open https://example.com/robots.txt on the production hostname, check the User-agent: * rules, look for an OAI-SearchBot group, and test whether article paths are covered by a Disallow rule. A broad Disallow: / can stop crawling, while an empty Disallow: allows it. Keep the search crawler’s policy separate from the training crawler’s policy, and remember that robots.txt controls access rather than guaranteeing removal from an index.
What can a five-minute robots.txt check tell you?
It can identify the most common configuration reason an AI search crawler cannot fetch your site: a rule that blocks the crawler itself or the paths containing your articles. The check does not prove that every visibility problem is caused by robots.txt. A page can be crawlable yet absent from an answer for reasons outside this file, and a disallowed URL can still be referenced elsewhere.
As an Amazon Associate I earn from qualifying purchases.
Robots.txt is a plain-text policy file at the root of a hostname. Google describes it as a file that tells crawlers which URLs they may access. It is a set of crawler instructions, not an authentication barrier, a deletion command, or a guarantee that a URL will never be indexed.
How do I run the five-minute check?
Use a production browser window, not a local development copy. The file you need to inspect is the one visitors and crawlers receive from the public hostname.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
- Open the live file. Go to
https://example.com/robots.txt, replacing the example hostname with the exact host serving your pages. Check the response after any CDN or edge layer, because that is what a crawler sees. - Locate the wildcard group. Find
User-agent: *. A line readingDisallow: /blocks crawling across the site for crawlers that follow the group. An emptyDisallow:line explicitly allows crawling under that group. - Find the OpenAI search group. Search the file for
OAI-SearchBot. OpenAI says this crawler is used to surface websites in ChatGPT search features. If your site opts out of it, pages will not be shown in ChatGPT search answers, although they may still appear as navigational links. - Inspect every article path. Look for rules such as
/blog/,/articles/, or wildcard patterns that match the directory or slug where the missing page lives. Path matching determines whether a rule reaches a particular URL; do not stop after checking only the homepage. - Re-open and monitor. After saving a change, load the public file again and verify that the intended text is actually being served. OpenAI’s current documentation says its search systems may take about 24 hours to adjust to a robots.txt update, so judge the result after that processing window rather than immediately.
Which robots.txt directives matter for AI search?
The useful directives here are the crawler identity, the paths it may access, and (where supported) a sitemap reference. The meaning depends on the group in which the directive appears.
User-agent
User-agent: * is a broad group for crawlers that accept the wildcard. A named group, such as User-agent: OAI-SearchBot, targets that crawler specifically. Keeping named groups separate makes it possible to express different policies for search discovery and other uses.
Disallow
Disallow: / matches the site from its root and is a site-wide crawl block. A directory rule such as Disallow: /blog/ targets URLs under that path, while a narrower path can target one section or pattern. An empty value, written Disallow: with nothing after the colon, means that the group has no disallowed path.
Allow
Allow: / states that the named crawler may fetch the site. It is especially clear in a crawler-specific group because it documents an intentional exception to a broader policy. Google supports user-agent, allow, disallow, and sitemap; directives outside the supported set should not be treated as portable controls.
Sitemap
A sitemap line can point crawlers to a list of canonical URLs, but it does not cancel a matching disallow rule. Treat discovery and permission as separate jobs: the sitemap advertises URLs, while robots.txt determines whether a crawler may fetch them.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
How do I allow AI search while blocking training crawlers?
OpenAI documents separate controls for OAI-SearchBot, which is used for search visibility, and GPTBot, which relates to potential training use. Because the settings are independent, a publisher can allow the search crawler while disallowing the training crawler, provided that policy matches its licensing and access goals.
Recommended Free Tools
A minimal example is:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This is an example policy, not a universal recommendation. Before publishing it, decide whether your agreements permit search use, training use, both, or neither. Check the live response after deployment; a CMS, redirect, CDN cache, or security layer can serve different text from the file you edited.
Do not assume that an instruction for one crawler automatically governs another. A policy naming GPTBot does not, by itself, grant or deny access to OAI-SearchBot. Conversely, allowing the search crawler does not grant training permission.
Will Disallow: / remove my pages from search?
No. Robots.txt controls whether a crawler may access a URL; it is not a guaranteed de-indexing mechanism. Google warns that a disallowed URL can still be indexed when other pages link to it, even though the crawler cannot fetch the content for a normal crawl.
Choose the control that matches the outcome you need:
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
- Stop a crawler fetching content: use a matching robots.txt rule.
- Keep a page out of an index: use an appropriate
noindexdirective where the crawler can see it, or use the search engine’s removal process. - Restrict access to people with permission: require authentication or another access control. Robots instructions are voluntary and do not protect confidential material.
A blocked URL cannot reliably communicate a noindex instruction to a crawler that is forbidden from fetching the page. If de-indexing is the goal, do not rely on a disallow rule alone.
What should I compare before choosing a control?
The right setting depends on whether you are managing visibility, training use, crawl access, or actual removal. These mechanisms solve different problems.
| Mechanism | Primary scope | What it controls | Important limitation |
|---|---|---|---|
OAI-SearchBot group |
OpenAI search crawler | Whether the site signals permission for search crawling and ChatGPT search visibility | OpenAI says opted-out sites will not be shown in ChatGPT search answers, though navigational links may still appear |
GPTBot group |
OpenAI crawler associated with potential training use | A separate training-use preference | It does not set the policy for OAI-SearchBot |
| Wildcard or path-specific robots rules | Any crawler that follows the matching group | Crawl access to the whole site, a directory, or a URL pattern | Compliance is voluntary, and a blocked URL may still be indexed |
noindex, authentication, or removal tools |
Indexing or human access | De-indexing intent or restricted access | These are different from crawl blocking; a crawler generally must fetch a page to read a page-level directive |
| WAF or bot-management controls | Network requests reaching your infrastructure | Technical enforcement of crawler-specific blocking | Configuration is outside robots.txt and can affect legitimate traffic if too broad |
Why might an article still be invisible after I allow OAI-SearchBot?
First, confirm that the allowance applies to the article’s actual path, not merely to the homepage. A separate wildcard group, a more specific directory rule, or a pattern can still prevent fetching the content. Review the complete file from top to bottom and test the exact URL path against each relevant rule.
Second, verify that you changed the production file. Editing a template in a staging environment, an old CMS setting, or a cached edge response will not change what the crawler receives. Re-open the public URL after deployment and compare the served text with the intended policy.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Third, allow time for the crawler’s configuration to update. OpenAI documents an adjustment period of about 24 hours after a robots.txt change. During that period, keep the policy stable so you can tell whether the new response was processed.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Finally, separate crawling from ranking or answer selection. Permission to fetch a page makes it eligible for discovery; it does not promise that every page will be selected for an answer. Robots.txt can explain an access failure, but it cannot guarantee placement.
How do voluntary rules differ from technical blocking?
Cloudflare’s documentation describes robots.txt compliance as voluntary. A cooperative crawler reads the file and follows its matching instructions; a hostile or misconfigured client can ignore them.
If the requirement is enforcement rather than signaling, use controls at the web-application firewall or bot-management layer. Those systems can identify and block requests technically, while robots.txt communicates your preferred policy to compliant crawlers. Keep the two layers aligned: a WAF block can make a crawler fail even when robots.txt allows it, and a robots disallow can prevent a compliant crawler even when the WAF would otherwise pass it.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Test enforcement carefully. Overly broad bot rules can block accessibility tools, monitoring services, or ordinary users who share infrastructure with automated traffic. A named crawler rule is narrower than a blanket denial, but every technical control should be reviewed against your site’s legitimate automation.
What are the most common robots.txt mistakes?
- Checking the wrong hostname: the file at a staging or alternate host does not describe the production host serving your articles.
- Leaving a site-wide block in place:
Disallow: /under a broad group is easy to overlook after a launch or migration. - Blocking the content directory: a rule for
/blog/,/articles/, or a wildcard can cover every page readers want discovered. - Combining unrelated policies: placing search and training preferences in one undifferentiated wildcard rule makes the intent unclear and can produce the wrong result.
- Expecting robots.txt to erase an indexed URL: crawling permission and index removal are separate jobs.
- Relying on unsupported directives: Google does not support fields such as
crawl-delay; including one does not create a Google crawl-rate policy. - Forgetting the post-edit verification: a deployment, redirect, cache, or CMS regeneration can replace the text you thought you published.
What is the safest way to change the file?
Record the current production file before editing so you can restore it. Make the smallest change that expresses the intended policy, preserve unrelated groups, and avoid deleting rules whose purpose you have not identified.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
- Write down the exact crawler and paths involved in the visibility problem.
- Adjust only the relevant group, such as adding an explicit
OAI-SearchBotallowance or removing an unintended article-directory block. - Validate that every directive is spelled correctly and that each group is separated clearly.
- Publish through the same system that serves the production hostname.
- Fetch the public file again, then monitor crawler and search behavior after the documented processing window.
If the business requirement changes from “let search discover this” to “keep this private,” replace the robots-only approach with authentication or another access-control system. If the requirement is “remove this from an index,” use a page-level noindex or removal workflow that can actually communicate that intent.
What should a publisher decide before publishing an AI crawler policy?
Make four decisions explicitly. First, decide whether search visibility is beneficial for your audience. Second, decide whether potential training use is licensed or permitted. Third, decide whether the policy is a preference for compliant crawlers or a technical block that must be enforced at the edge. Fourth, decide whether any existing URLs must be removed from an index rather than merely kept from future crawling.
Document those decisions beside your deployment configuration. A short comment in the CMS or change log can prevent a future migration from restoring Disallow: / or merging independent crawler policies into a single wildcard group. Revisit the file whenever your content directories, licensing terms, or security controls change.
Frequently Asked Questions
Can robots.txt protect a private admin page?
No. It only asks compliant crawlers not to fetch a path, while an unauthorized person or bot can ignore the request. Put administrative and confidential content behind authentication or another access-control mechanism.
Does allowing OAI-SearchBot automatically allow GPTBot?
No. OpenAI documents these as independent crawler controls. An explicit allowance for search does not grant permission for potential training use, so each crawler needs its own policy decision.
Can I use crawl-delay to control Google after opening AI search access?
Google does not support the crawl-delay field. Do not expect that line to change Google’s crawling behavior; use supported directives and, when necessary, infrastructure-level rate or bot controls.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsWhat if my CMS keeps restoring the old robots.txt rules?
Treat the public response as the source of truth. After each deployment, open the production file and compare it with the intended groups; a template, cache, redirect, or regenerated setting can silently replace a manual edit.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
