Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Robots.txt cannot legally or technically stop every AI training crawler by itself. It is a machine-readable request that compliant crawlers are expected to follow, not a password, firewall, or other access lock. The IETF’s RFC 9309 states explicitly that robots.txt rules “are not a form of access authorization.” Whether ignoring a rule creates liability depends on the jurisdiction, who owns the material, how access was obtained, the wording of any rights reservation, and what the collected data is used for.
Use robots.txt as one layer of a policy: identify which kinds of crawling you want, publish precise rules for documented bots, and add authentication, paywalls, request filtering, or managed bot controls when preventing retrieval is important. Keep evidence of your policy and requests, because a technical block and a legal claim are separate questions.
As an Amazon Associate I earn from qualifying purchases.
What does robots.txt actually do?
Robots.txt is a plain-text file normally served at https://example.com/robots.txt. It groups instructions by crawler user-agent and says which URL paths a crawler is requested to fetch or avoid. Under RFC 9309, these are voluntary rules for crawlers; they do not authenticate a visitor, reject an HTTP request, encrypt a page, or create a technical barrier.
A compliant bot reads the file before crawling and may skip a path listed under Disallow. A noncompliant bot can request the same URL anyway. The file is public, so it also reveals your preferences and should not be used to hide secrets. Anything that must remain confidential belongs behind authentication or another access control.
Popular on Amazon
- 【90 Degree Full Function USB C Adapter】Supports the latest USB4 standard for up to 40Gbps of bandwidth, 8K video support, and up to 240W Fast charging; Support USB PD 3.1 Extended Power Range (EPR) including 140W, 180W and 240W. Your productivity is easily doubled. *Warm Tip: For the best fit, please make sure the USB-C port is fully exposed; thick or bulky cases may affect compatibility.
- 【Reliable Quality】Made from high-quality, scratch-resistant aluminum alloy, delicate metal frame. get excellent wear resistance and heat dissipation. Non-Slip Design for easier insertion and removal.
- 【Small and portable】Ultra thin size, ultra compact design, a petite physique that matches the minimalist design of your modern laptops. Coming with a slim figure that won't block the port next to the one it's plugged into.
- 【Compatibility】Widely work with most usb c, thunderbolt 4/3 devices. Compatibilit for ROG Ally, Steam Deck, Switch, MacBook Pro, Quest 3 VR headset, Surface Book 2, iPhone 18/17/16/15, Samsung Galaxy S23/S22/S21, Note 20/10, Google Pixel, ChromeBook, etc.
- 【Excellence Customer Support】If you experience any issues or cable failure, simply reach out to our dedicated customer service team. We provide a prompt response within 18 hours, providing you with the necessary technical support to resolve any concerns.
A minimal example
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Disallow: /admin/
This example asks GPTBot not to crawl the site, allows OAI-SearchBot to fetch it, and keeps the administrative path out of the ordinary crawler group. The result depends on each bot recognizing the exact token and honoring the protocol. It does not prevent a different user-agent from requesting the pages.
Can a robots.txt disallow create legal liability when a crawler ignores it?
Possibly, but there is no global yes-or-no rule. A robots.txt instruction can be evidence of the owner’s wishes and, in some legal systems, a machine-readable rights reservation can matter to text-and-data-mining rules. It does not automatically turn every ignored request into copyright infringement, breach of contract, computer misuse, or circumvention.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11The relevant facts usually include:
- Which country’s law applies to the collection and later use.
- Who owns or controls the copyright and other rights in the material.
- Whether the material was publicly accessible or protected by a login, paywall, or other barrier.
- Whether the site’s terms contain enforceable restrictions and whether the crawler agreed to them.
- What the data was used for: indexing, search answers, model training, fine-tuning, evaluation, or another purpose.
- Whether the jurisdiction recognizes a machine-readable reservation for the activity at issue.
Consequently, publishing a disallow is sensible notice, but it is not a universal legal veto over all collection or all AI uses. A lawyer in the relevant jurisdiction can assess a particular dispute; this article cannot determine whether a claim would succeed.
What did the US court decide in Ziff Davis v. OpenAI?
The Southern District of New York’s 2025 decision is important but narrow. In discussing a pleaded claim under section 1201 of the Digital Millennium Copyright Act, the court held that the complaint did not adequately allege that robots.txt effectively controlled access or that ignoring the file was affirmative circumvention. The court compared a robots.txt request to a sign asking visitors to “keep off the grass”: a request is not the same as an effective access control.
Read the ruling in its procedural context in Document 300. It does not establish that every scraping, copyright, contract, trespass, or state-law claim fails whenever a site has a robots.txt file. It also does not decide how another court or another set of facts would treat a paywall, authenticated API, rate limit, or explicit contractual restriction.
Also popular on Amazon
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
How does EU law treat machine-readable reservations?
EU law gives the issue more specific text-and-data-mining context, while still not turning robots.txt into an access-control system. A 2025 European Parliament commissioned study explains that the traditional Robots Exclusion Protocol is voluntary and is not generally a binding contract merely because a file exists. It also discusses Article 4(3) of the Copyright in the Digital Single Market Directive, under which rights holders may reserve text-and-data-mining rights in an appropriate manner, including machine-readable means for publicly available online content.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
That distinction matters. A reservation may be relevant when deciding whether an exception applies, but it does not retroactively block a request, identify the correct rights holder, or settle whether a particular model-training act is lawful. The wording, implementation, type of work, applicable exception, and use all remain important.
The EU AI Act adds obligations for general-purpose AI model providers. The European Parliament study and the UK government’s 2026 report describe requirements for a copyright-compliance policy and sufficiently detailed summaries of training content. Those transparency and policy duties are different from a technical rule that prevents a crawler from downloading a page. The Bundesnetzagentur’s Code of Practice publication records signatory commitments to recognize robots.txt and other suitable machine-readable reservations; treat those as commitments by participants, not as a replacement for the underlying copyright rules.
What is the UK position?
The UK Intellectual Property Office and Department for Science report published in 2026 says that, at publication, UK law did not generally require AI developers to publicly disclose which copyright works trained a model or how those works were obtained. The statement describes the position at the time of the report, not a permanent answer to every future legislative or court development.
The same UK report discusses practical controls and the EU’s separate requirements. UK site owners should therefore distinguish current legal duties from policy proposals, voluntary commitments, and evidence that a developer ignored a stated preference.
Free tools Windows power users keep installed
One-click scans. No signup required.
Also popular on Amazon
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Which AI crawlers should a site owner distinguish?
Do not treat “AI bot” as one category. A provider can use separate crawlers for model training, search-result discovery, and a user’s one-off request, with different settings and implications.
Training versus search retrieval
OpenAI’s crawler documentation identifies GPTBot as a crawler for content that may be used to train generative AI foundation models. It separately identifies OAI-SearchBot as the crawler that surfaces sites in ChatGPT search and says the settings are independent. A site can therefore allow search visibility while disallowing GPTBot, if that matches its policy.
The documentation also describes ChatGPT-User, which responds to a user-initiated request. A rule aimed at training does not necessarily answer whether a page should be fetched to satisfy an individual user. Check the current documentation and the exact user-agent token before changing your file.
Google’s crawler categories
Google’s crawler overview says its common crawlers follow robots.txt for automatic crawling, while distinguishing special-purpose crawlers and user-triggered fetchers. The practical lesson is the same: identify the particular crawler and purpose instead of assuming that one generic setting covers every fetch.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →How should I write and deploy a robots.txt policy?
Start with the outcome you want, then implement the narrowest rule that expresses it. Blocking all automated access can remove ordinary search indexing and referrals along with training crawls.
Also popular on Amazon
- Portable and powerful USB-C HUB: BENFEI USB Type-C HUB, with super-soft and knot-free silicone woven design cable, meets most mobile office needs. Compact, lightweight, stylish, and powerful portable USB C Hub equipped with 1 x HDMI port, 1 x 100W charging, and 3 x USB ports. 18-month warranty, 24-hour response, to ensure you feel at ease when using our product.
- Design centered on comfort and reliability: Thanks to BENFEI's end-to-end in-house cable production capability, in-house PCBA and assembly capability, using the industry's most advanced silicone woven design and process, 20cm cable in length, no knots, super-soft, the HUB is easy to use in all scenarios: laptop, tablet, stand etc. Super-soft, 25000+ life cycles, to meet your daily carrying and office needs.
- 100W Charging: Support up to 90W USB C pass-through charging via Type-C port to keep your laptop powered. 10W is reserved for other interface operations. No data and video function on the Type-C port.
- 4K HDMI Display: The HDMI port supports media display at resolutions up to 4K 30Hz, keeping every incredible moment detailed and ultra vivid. Please note that the C port of the Host device needs to support video output.
- Transfer Files in Seconds: Transfer files and from your laptop at speeds up to 10 Gbps with USB A 3.2 port. Extra 2 USB A 2.0 ports are perfectly for your keyboards and mouse.
- Separate your goals. Decide independently whether you want conventional search indexing, AI search referrals, model training, fine-tuning or evaluation collection, and user-triggered retrieval. Record which outcomes are allowed.
- Read each provider’s current documentation. Confirm the exact user-agent string, stated purpose, and any published IP-verification method. A forged user-agent can imitate a compliant bot, so identity checks may be necessary for enforcement.
- Create specific groups. Put a documented crawler token in its own group when you want a different rule from
User-agent: *. Avoid broad patterns that unintentionally block search engines or site functions. - Publish at the site root. Serve the file at the origin’s
/robots.txtURL, with a normal successful response and valid plain text. Rules in a subdirectory do not govern the whole host. - Check the matching behavior. Review path syntax, overlapping groups, and any
Allowexceptions. Test that the intended crawler sees the intended group and that ordinary search crawlers are not accidentally disallowed. - Keep dated records. Save each published version, the relevant provider documentation, server logs, terms of use, and any opt-out or takedown correspondence. Records can show what notice was given and when, even though they do not guarantee a legal result.
- Review regularly. Crawler names, purposes, verification ranges, and provider policies change. The Internet Architecture Board’s RFC 9969 workshop report describes continuing variance among vendors and the difficulty of mapping a crawl-time instruction to later training or inference uses.
What technical controls are stronger than robots.txt?
If the goal is to prevent a request from succeeding, use an actual access control in addition to a preference file. Options include:
- Authentication: Require an account, API key, signed URL, or other credential before serving the content.
- Paywalls and membership gates: Keep protected material behind a server-side entitlement check rather than relying on a front-end overlay.
- Request filtering: Apply rate limits, network and ASN rules, behavioral detection, and verified-bot allowlists at the web server, CDN, or security layer.
- Managed bot controls: Security services can classify traffic and, in some deployments, distinguish lifecycle stages such as training, fine-tuning, and inference. The UK report discusses such controls, including examples from major edge-security providers.
- Content minimization: Do not publish confidential or high-value source material openly when it needs stronger protection. Technical barriers work best when the origin server is not separately exposed.
These measures raise the practical cost of collection and can stop many requests, but they do not decide whether earlier collection was lawful or whether a model may use material obtained elsewhere. Keep logs and preserve evidence of blocked and allowed requests.
What trade-offs should a site owner consider?
Every rule should be evaluated on four axes:
| Decision | Question to ask | Typical trade-off |
|---|---|---|
| Crawler purpose | Is this training, search discovery, or a user-triggered fetch? | Blocking training may still allow search referrals; blocking all bots can reduce visibility. |
| Enforcement strength | Is a voluntary signal enough, or must requests fail? | Robots.txt is easy to deploy but nonbinding technically; authentication and filtering are stronger but add friction and cost. |
| Discoverability | Do search engines and answer services need access? | A broad disallow can remove pages from indexes and referral channels. |
| Legal context | Which law, rights owner, exception, and use apply? | A machine-readable reservation may matter in one framework without resolving every claim or jurisdiction. |
If you host articles, images, code, or datasets supplied by others, confirm who controls the relevant rights. Your site’s preference may not be identical to the copyright owner’s instruction, and you may need authority before making a reservation or granting permission.
What should I do if a bot ignores my rule?
First verify that the request came from the crawler you intended to address. Check DNS or published IP ranges where the provider offers verification, inspect timestamps and paths in server logs, and preserve the file version that was live at the time. A spoofed user-agent or an unrelated scraper may require a different response.
Next, choose a proportionate technical action: rate-limit, challenge, block the verified source, require authentication, or remove the public endpoint. Send a clear written notice through the provider’s published contact route and keep the response. If valuable works, confidential data, or significant business impact are involved, obtain jurisdiction-specific legal advice before asserting copyright, contract, computer-misuse, or circumvention claims.
Also popular on Amazon
- 【Read before You Buy】 Note: This adapter is USB 2.0 version and cannot transmit HDM, VGA, or other video signals. Nevertheless, they are fully capable of charging devices,duplicating files, and can be seamlessly integrated with USB-C earphones, wireless network adapters, and SD card adapters.
- 【Easy to Use and Safe】Thanks to modern manufacturing techniques, we've made these adapters sufficiently compact so that they can remain inserted in your device without noticeable protrusion or inconvenience. This means you don't have to constantly plug and unplug them, reducing the likelihood of wear and misalignment, and making them more convenient to carry around.
- 【Reliable Quality】Engineered for enhanced durability, durable aluminum alloy shell material,get excellent wear resistance and heat dissipation,the male USB C and female USB connectors & the male USB and female USB C connectors are designed to be plugged and unplugged up to 20000 times. Provides data transfer, device charging, memory space expansion, and additional computer peripherals.
- 【Think more for you】Do you need to connect a USB-C-enabled device to an odd device that only has USB-A ports? Then these USB-A male to USB-C female adapters are made for you.Simply insert the adapter into the USB-A port on the device, and you can instantly connect your USB-C device to the device that originally didn't have the correct connection port.
- 【Trusted xiwxi Quality and Service】As a responsible brand seller, you will receive xiwxi's 12-month Quality assurance and lifetime friendly customer service, you're not just buying a product,you're investing in peace of mind with all-round quality and exceptional customer service.
Do not assume that a block proves a legal violation, or that an ignored robots.txt file alone proves one. The strongest record combines ownership information, the exact reservation, evidence of access, the crawler’s identity, the use you are challenging, and the applicable law.
How we reached the practical guidance
The recommendations above follow the technical scope in RFC 9309, the jurisdictional distinctions in the European Parliament study and the 2026 UK report, the narrow holding in Ziff Davis, and the crawler-purpose documentation published by OpenAI and Google. They do not rely on a claimed prevalence or success rate: no verified statistic establishes how often AI-training crawlers honor opt-outs.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Frequently Asked Questions
Does adding Disallow: / make my website private?
No. It asks compliant crawlers not to fetch the site, but the file is public and any client can request the URLs. Use authentication, a paywall, or server-side filtering for private material.
Can I allow AI search while refusing model training?
Often, yes, when the provider documents separate crawlers and independent controls. For example, OpenAI documents separate tokens for GPTBot and OAI-SearchBot; verify the current documentation and keep the groups separate.
Should I remove pages from search engines to avoid AI training?
Not necessarily. Search indexing, AI search retrieval, and training can be separate activities, so a blanket block may sacrifice discoverability without addressing every possible collection route. Decide which outcome matters before changing rules.
Does an EU robots.txt reservation guarantee that a model provider broke copyright law?
No. It may be relevant to the text-and-data-mining framework, but the result still depends on the work, rights holder, wording, use, applicable exception, and other facts. A reservation is evidence of a preference or rights position, not an automatic court judgment.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errors

