Yes, but only in a limited, provider-specific way. A publisher can tell OpenAI’s GPTBot not to crawl pages for potential training, use Google-Extended to control whether content helps improve Gemini and Vertex AI generative APIs, and separate search visibility from training with OAI-SearchBot and noindex. These are machine-readable preferences that depend on crawler cooperation. They are not a universal block, a copyright licence, or a guarantee that every AI developer will exclude the material.
A negotiated licence is different: it is an affirmative contract that can define which archive or feed may be used, for what purpose, for how long, with what attribution and payment. The strongest practical policy combines per-provider crawler rules, access controls, rights metadata and contracts for high-value content.
As an Amazon Associate I earn from qualifying purchases.
What an AI opt-out can—and cannot—do
“Opt out of AI training” sounds like one switch, but publishers are making several separate decisions:
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minute- Training: whether a provider may use pages to improve a model.
- Search discovery: whether a service may find, retrieve and cite a page in an answer.
- Index display: whether a link or title may be surfaced.
- Downstream use: whether a provider may use an archive, feed or licensed database in products beyond ordinary crawling.
Each decision can have a different crawler, directive or contract. Blocking one user-agent does not automatically block another provider, and a crawler directive cannot rewrite data that was already copied by someone else. It also does not settle copyright questions; legal outcomes depend on jurisdiction and facts.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
OpenAI’s three relevant controls
OpenAI’s publisher guidance distinguishes GPTBot from OAI-SearchBot. OpenAI says: “Publishers should disallow the GPTBot user-agent from sites and pages they wish to exclude from potential training.” The guidance also says that pages opting out of GPTBot will not be trained on when users opt in to training.
OAI-SearchBot serves a different purpose: it helps ChatGPT discover pages that can be cited. A publisher may allow OAI-SearchBot while disallowing GPTBot. If crawling is allowed but a page carries a noindex directive, the page can be kept out of link and title surfacing; the crawler must be permitted to read the tag for that instruction to work.
Google-Extended is provider-specific
Google introduced Google-Extended on September 28, 2023, as a robots.txt control for managing whether sites help improve Bard and Vertex AI generative APIs, including future generations of the models powering those products. It is not a universal “no AI” setting and does not replace Google’s ordinary search controls.
Free tools Windows power users keep installed
One-click scans. No signup required.
How the controls compare
| Control | Primary purpose | Typical scope | Enforceability | Commercial upside |
|---|---|---|---|---|
| GPTBot directive | Exclude pages from OpenAI’s potential training crawl | OpenAI user-agent; site or selected paths | Voluntary crawler compliance; not a security barrier | None by itself |
| OAI-SearchBot directive | Permit or restrict ChatGPT discovery and citations | OpenAI search crawler; site or paths | Voluntary signal | Possible referral and citation visibility when allowed |
noindex |
Prevent eligible links and titles from being surfaced | Individual URL or response header | Requires the crawler to fetch and honor it | None by itself |
| Google-Extended directive | Manage use of content to improve Gemini and Vertex AI generative APIs | Google-Extended user-agent; site or paths | Voluntary signal | None by itself |
| C2PA and related rights metadata | Carry ownership, licence and usage preferences with an asset | Asset or manifest fields | Depends on adoption and preservation of manifests | Can support clearer permissions and provenance |
| Negotiated licence | Authorize defined uses in exchange for agreed terms | Contract-defined archive, feed, assets or products | Contractual; remedies depend on the agreement and applicable law | Payment, attribution, access, expertise or product collaboration |
There is no reliable universal percentage of publishers opting out, standard licence price or guaranteed robots.txt success rate. A 2026 FT Strategies snapshot recorded 48 publicly announced publisher-company relationship records covering 30 publisher domains, but it warned that crawler observations do not establish contractual rights, licensing status, compliance, legal entitlement or infringement.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Using robots.txt for OpenAI and Google
Robots.txt is a plain-text file at the root of a domain, normally served as /robots.txt. It expresses instructions to named user-agents. It is useful for communicating policy at scale, but any crawler can ignore it, a copied page can circulate elsewhere, and it does not authenticate a requester.
Exclude GPTBot from an entire site
User-agent: GPTBot
Disallow: /This expresses the preference to exclude the whole site from GPTBot’s potential training crawl. To target only a section, replace the slash with a path, such as Disallow: /premium/. Review whether your platform generates alternate URLs, image paths or API endpoints that require separate rules.
Allow ChatGPT discovery while restricting training
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /This combination separates the two decisions: GPTBot is disallowed, while OAI-SearchBot may retrieve pages for discovery and citation. A page-level noindex can still prevent link and title surfacing:
<meta name="robots" content="noindex">Use the tag only where the crawler can access the HTML. Blocking the crawler in robots.txt prevents it from seeing a noindex instruction, so do not treat those mechanisms as interchangeable.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Restrict Google-Extended
User-agent: Google-Extended
Disallow: /This tells Google-Extended not to use the site under that user-agent for improving the covered generative-AI systems. It does not automatically remove pages from Google Search, and it does not instruct unrelated AI crawlers.
Deploy and verify carefully
- Inventory paths first. Separate public news, paywalled articles, image files, feeds, APIs, archives and partner content. A single site-wide rule may be too broad.
- Check the exact user-agent. GPTBot, OAI-SearchBot and Google-Extended are different identities. A rule for one does not govern the others.
- Test the served file. Confirm that the production domain returns the intended text at
/robots.txt, without a cache serving an older version. - Inspect status codes and redirects. A reliable 200 response is easier for crawlers to process than a blocked, redirected or intermittently failing file.
- Monitor logs. Record user-agent, IP, path, response and timestamp. Treat user-agent strings as claims to verify, not proof of identity.
- Review after site changes. New subdomains, content-delivery hosts and syndication feeds may need their own policy.
Does blocking GPTBot remove a site from ChatGPT search?
Not necessarily. GPTBot and OAI-SearchBot represent different functions. A publisher can disallow GPTBot while allowing OAI-SearchBot, preserving the possibility of discovery and citations. Conversely, blocking OAI-SearchBot can reduce ChatGPT’s ability to find and cite current pages, even if a separate training decision is unchanged.
noindex adds another layer. When a permitted crawler reads it, the instruction can stop a link or title from being surfaced. It is therefore possible to allow retrieval for a technical purpose while declining search-result display, but the outcome depends on the provider honoring the directive.
What rights metadata such as C2PA adds
Robots.txt describes a crawler preference at the site or path level. Rights metadata can travel with an individual image, video or document and express more detailed provenance and usage information. C2PA guidance includes fields such as copyrightHolder, copyrightNotice, copyrightYear, license, usageInfo and c2pa.ai_training.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
The European Commission’s AI-copyright consultation lists TDMRep, C2PA TDM assertions, AI.txt, Do not train registry, JPEG Trust, TDM.ai and Open Rights Data Exchange among protocols under discussion. These approaches can identify an owner, point to a licence, or state a training preference at finer granularity than a domain-wide file.
Why metadata is not a guaranteed lock
- A platform must recognize the protocol and interpret the particular field.
- The manifest must remain attached to the asset through copying, resizing, syndication and storage.
- Different file formats and delivery systems may preserve metadata inconsistently.
- Metadata records a claim or permission; it does not, by itself, create a negotiated contract or prevent an unauthorized copy.
Use rights metadata as part of provenance and rights management, not as a substitute for access controls or a written agreement.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Why a licence is stronger than an opt-out
A licence is affirmative permission. It can define the exact archive or feed, permitted model-training and retrieval uses, attribution wording, geographic reach, term, payment, security obligations, audit rights, deletion or renewal rules, and what happens when the relationship ends. Robots.txt supplies none of those terms.
Recommended Free Tools
What recent deals illustrate
The Financial Times/OpenAI announcement described an agreement covering attributed Financial Times content in ChatGPT, improving model usefulness and collaboration on new products and features. The Associated Press said its arrangement licensed part of its text archive to OpenAI while AP received OpenAI technology and product expertise. Those announcements illustrate different negotiated benefits; they do not establish a standard market price or template for every publisher.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Questions to settle before signing
- Which works, dates, editions, languages and feeds are included?
- Is use limited to retrieval, evaluation, fine-tuning, pretraining, or all of these?
- May the provider retain copies, and for how long?
- How will attribution appear in answers, interfaces and derivative products?
- Are synthetic outputs, embeddings, summaries or customer tools covered?
- What reporting, payment formula, audit and breach remedies apply?
- Can the publisher suspend future ingestion without terminating unrelated services?
Have counsel review the agreement for the jurisdictions in which the publisher and provider operate. A licence can coexist with crawler rules: a publisher may license a curated archive while disallowing general web crawling of the rest of the site.
A practical layered policy for publishers
- Classify content by value and sensitivity. Mark breaking news, evergreen explainers, photographs, subscriber-only work, licensed third-party material and public-domain material separately.
- Choose the desired outcome per class. Decide whether each class should be available for search citation, excluded from potential training, hidden from indexing, or offered only under contract.
- Set provider-specific directives. Configure GPTBot, OAI-SearchBot and Google-Extended independently; do not assume a wildcard captures every future crawler.
- Use page and response controls. Apply
noindexwhere link/title surfacing is unwanted, and use authentication, paywalls or signed feeds for genuinely restricted material. Robots.txt alone is not access control. - Attach provenance and rights metadata. Preserve C2PA manifests and licence information through publishing and syndication workflows where your formats support them.
- Offer controlled machine access. If a provider needs a reliable feed, create a documented endpoint with rate limits, attribution requirements and contractual terms rather than relying on anonymous scraping.
- Negotiate for high-value archives. A contract is the route to payment, defined retention, attribution and product commitments.
- Keep an evidence trail. Version robots.txt, retain deployment logs, document licence scope and record complaints or suspected non-compliance.
Common mistakes and fixes
Treating robots.txt as a legal guarantee
Mistake: assuming one disallow line binds every AI company or proves that content was not used. Fix: describe it internally as a crawler preference, then add technical access controls, metadata and contractual enforcement where the risk warrants it.
Using one rule for search and training
Mistake: blocking GPTBot and expecting ChatGPT citations to behave the same way, or allowing a search crawler and assuming training permission. Fix: make separate decisions for GPTBot, OAI-SearchBot, noindex and Google-Extended.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Blocking a crawler but exposing the same data elsewhere
Mistake: disallowing an article path while leaving an unrestricted RSS feed, API, CDN host or syndication partner. Fix: inventory duplicate delivery routes and apply policy consistently.
Assuming metadata survives distribution
Mistake: placing a C2PA assertion on the original file and assuming every resized or reposted copy retains it. Fix: test your publishing pipeline and treat manifest preservation as an operational requirement.
Inferring a licence from a crawler observation
Mistake: concluding that a publisher has or has not licensed content because a crawler appears in logs or robots.txt. Fix: rely on the actual public announcement or contract; crawler behavior does not establish contractual rights, licensing status or infringement.
Choosing the right control
| If your priority is… | Best starting control | What it does not solve |
|---|---|---|
| Reduce potential training crawl by OpenAI | Disallow GPTBot on selected paths | Other providers, copied data, legal status or existing datasets |
| Keep possible ChatGPT citations | Allow OAI-SearchBot; use noindex only where display should be withheld | Model-training permission and guaranteed citation frequency |
| Limit Google generative-AI improvement use | Disallow Google-Extended | Other AI systems and ordinary Google Search indexing |
| Express asset-level ownership and usage terms | C2PA or another adopted rights-metadata system | Adoption, manifest retention and contractual enforcement |
| Secure payment, attribution and defined archive use | Negotiate a licence | Unlicensed third-party copies outside the contract |
The practical answer is therefore neither “robots.txt stops AI” nor “opt-outs are useless.” They are useful signals when matched to the correct provider and objective. For valuable or sensitive material, combine them with access controls, preserved rights metadata, monitoring and a licence that states exactly what the provider may do.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
