In a one-day census of 185 selected websites, Pennyforge reported that 35 (18.9%) published a hard opt-out for AI crawlers. The share was highest among media and publishers (20 of 30) and lowest among government portals (1 of 24). Every reported opt-out was a site-wide Disallow: / rule in robots.txt—a published instruction, not proof that a crawler was technically blocked.
What the 185-site census found
Pennyforge checked the sites on October 1, 2026, in a single pass from 17:15 to 17:22 UTC. The panel was a defined collection, not a random sample of European websites, so its percentages describe those 185 sites rather than Europe as a whole.
As an Amazon Associate I earn from qualifying purchases.
Of the 185 sites, 35 (18.9%) had what the author classified as a hard AI-crawler opt-out. All 35 used a full-site Disallow: /; the census reported no partial path-level opt-outs. Pennyforge notes that its per-user-agent parser did not fully implement RFC longest-path-match precedence, a limitation that could affect at most the one partial case it found.
Free tools Windows power users keep installed
One-click scans. No signup required.
Which sectors had the most opt-outs?
The largest contrast was between media and publishers and government portals. The figures below are Pennyforge’s reported results for its panel, not population estimates.
#1 Best Overall
| Panel group | Sites with a hard opt-out | Share |
|---|---|---|
| EU media and publishers | 20 of 30 | 66.7% |
| Dutch public-interest sites | 2 of 8 | 25% |
| Major Netherlands and Germany retailers | 4 of 23 | 17.4% |
| Technology, AI, and scholarly-infrastructure sites | 4 of 34 | 11.8% |
| Additional EU retail shops | 4 of 66 | 6.1% |
| EU and national government portals | 1 of 24 | 4.2% |
The panel consisted of those six groups: 23 major NL/DE retailers, 8 Dutch public-interest sites retained from an earlier slice, 66 additional EU retail shops, 24 government portals, 30 media and publisher sites, and 34 technology, AI, and scholarly-infrastructure sites. Pennyforge describes the panel as brand-biased and non-random; it cannot establish sector-wide or Europe-wide prevalence, or explain why an individual site chose a rule.
What the robots.txt checks showed
Pennyforge checked robots.txt for 12 user-agent names: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent, and MistralAI-SearchBot.
Rank #2
- File availability: 121 of 185 sites (65.4%) served a
robots.txtfile. Of those 121 files, 77 (64%) had no entries for any of the 12 listed AI crawlers. - Unavailable responses: 39 sites (21%) returned HTTP 403 for the file. Pennyforge treated those responses as temporarily unobtainable policy, not as evidence that no policy existed.
- Rules by crawler: CCBot appeared in 26 disallows, with no explicit allows in the panel. Reported disallow counts were GPTBot 22, ClaudeBot 18, Google-Extended 17, Bytespider 17, and Meta-ExternalAgent 16. MistralAI-SearchBot was not mentioned in any rule in the panel.
- Explicit allows: 9 of 185 sites had an explicit
Allowfor at least one listed AI user agent; Pennyforge said most named examples were consumer-electronics retailers.
These crawler-specific counts are Pennyforge’s reported parser results, not independently reproduced measurements. A missing entry in the checked file is not the same as an explicit permission, and a stated rule does not establish what a server will do when a crawler requests a page.
Recommended Free Tools
Did sites use a policy endpoint beyond robots.txt?
Pennyforge also probed /.well-known/ai-crawler. Twenty sites returned HTTP 200, but all 20 responses were HTML catch-alls rather than policy files; the census found no actual file at that path on any of the 185 sites. The check considered content type so a single-page-app fallback would not be mistaken for a published endpoint.
Rank #3
This result is limited to that URL and that panel on October 1, 2026. It does not establish whether other policy mechanisms exist elsewhere or whether adoption has changed since the crawl.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What “blocking” means here—and what it does not
A robots.txt disallow is a published request to compliant crawlers, not a technical access control. It does not by itself prove that every crawler obeys the instruction or that the site prevents access. As Pennyforge put it: “The de facto standard for TDM opt-out in Europe is a fragmented, voluntary robots.txt with per-UA lines — enforced only if the crawler chooses to check.”
Published rules and observed server responses are separate dimensions. CrawlIndex describes an index that measures both, but its dataset is distinct from Pennyforge’s 185-site census and should not be combined with these figures.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteThe census is a dated snapshot from one pass, not a measure of current settings or a time series. Its strongest supported finding is the contrast within its own panel: media and publisher sites published full-site opt-outs much more often than government portals. It cannot show the reason for that difference or establish how common such rules are across all European websites.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
