In a September 2026 snapshot of 70 usable robots.txt files from selected well-known domains, most files did not name an AI crawler. Among the files that did, blocking was common—but the results describe written rules for a hand-picked sample, not the web as a whole or what crawlers actually did.
What the 70-file snapshot found
SerpPrism maintainer Hongtao Ren fetched robots.txt files for 78 well-known domains on September 23, 2026, and analyzed 70 usable files against 22 selected AI crawler tokens. The domains spanned areas including news, commerce, SaaS, developer tools, education, and government; they were not a random sample. Eight responses could not be used because they were unreadable or not robots files.
As an Amazon Associate I earn from qualifying purchases.
- 44 of the 70 usable files named no AI crawler token; 26 named at least one.
- Of those 26 files, 22 blocked at least one crawler they named, and 13 blocked every named crawler.
- Four named crawlers but blocked none: Cloudflare, Netlify, Moz, and Twitch. Twitch’s empty Amazonbot directive allows everything; it is not a meaningful block.
These are counts from SerpPrism’s selected sample, not estimates of how common the policies are across all websites. The study evaluated whether each selected crawler could fetch the homepage under the file’s rules. It used a crawler-specific group when present, otherwise the wildcard group, and applied longest-match path rules with an equal-length Allow taking precedence. SerpPrism’s study provides the full account.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallWhich named crawlers were blocked most often?
The figures below count only the sample sites that named each token. A block rate in a small, selected group should not be read as a web-wide rate.
#1 Best Overall
- 【Powerful Load-bearing】12U Network Rack Open Frame is constructed from durable cold rolled steel; Rack shelf supports enhance stability, wall-mounted capacity of 130lbs, the ground-mounted up to 260lbs
- 【Considerate Designs】Open-frame layout, including a top panel adding space, anti-slip shelf stops fixing devices and compatible racks for stack and expansion to meet requirements of home server rack
- 【Complete Accessories】A 12U open frame server rack, two ventilated shelves, four shelf stops, four velcro straps and a set of equipment mounting screws
- 【Versatile Application】Ideal for space-efficient multi-device setups in warehouses, retail, classrooms, offices and more; Excellent choices as AV Rack/IT Rack
- 【Effortless Setup】 Network Rack includes hardware, a comprehensive manual, mounting hole drilling template and an online assembly video to simplify setup
| Crawler token | Named by | Blocked by | Share blocked among sites naming it |
|---|---|---|---|
| GPTBot | 17 sites | 10 | 59% |
| ClaudeBot | 18 sites | 13 | 72% |
| CCBot | 15 sites | 13 | 87% |
| Bytespider | 11 sites | 11 | All 11 |
| Applebot-Extended | 10 sites | 10 | All 10 |
| omgili | 9 sites | 9 | All 9 |
| Diffbot | 8 sites | 8 | All 8 |
SerpPrism reported these counts for its September 2026 snapshot. The all-blocked results apply only to the sites in the sample that named those tokens.
Why silence in robots.txt does not settle access
Of the 44 files that named no AI crawler, 42 allowed crawlers through their wildcard policy. Reddit and Pinterest instead had a wildcard Disallow: /, which blocks all crawlers covered by that group. So an absent AI-specific rule does not, by itself, tell you whether an unnamed crawler can fetch a page: its effective policy may come from the wildcard group.
Rank #2
- Adjustable Depth: 23-40'' adjustable depth is used for servers and network equipment, ensuring enough space for AV equipment, components, and cabling, while allowing you to access ports and equipment from multiple sides.
- Strong Load Capacity: Ground-Mounted Load Capacity: 500 lbs, Wall-Mounted Load Capacity: 150 lbs. The av rack is made of carbon steel for better weldability performance and can help save space while meeting your need to place multiple devices.
- User-friendly Design: Ergonomic design makes the open frame av rack easier to use. The additional top panel is able to place other items with more available space. Roller design moves anywhere and anytime, is convenient, and is more energy-saving.
- Complete Accessories: We provide the accessories you need, including 2 x Pallets, 145 x M5*10 Cross Head Screws, 4 x Casters, 4 x M10*50 Expansion Screws,10 x M6*12 Cage Nuts, 1 x Grounding Wire, 1 x User Manual.
- Wide Application: The server rack wall mount maximizes the use of available space, suitable for retail venues, classrooms, offices, and other places where space is limited.
Robots.txt is an advisory request, not an access-control mechanism. As Ren puts it, “robots.txt is a request, not a lock”. A file does not show whether a crawler complies, nor whether a site separately restricts requests through firewalls, rate limits, or other controls.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Policies can distinguish purpose, paths, and crawler groups
The sample’s examples show why a single “AI: allow” or “AI: block” label can obscure important differences. A policy may treat training-oriented crawlers differently from search crawlers or crawlers used for user-triggered retrieval. It may also allow some paths while restricting others, or put several tokens under one shared group.
Rank #3
- Adjustable Depth: Depth adjustable from 23" to 40", this open frame server rack accommodates servers and network equipment while providing ample space for A/V gears and cable management. Enjoy easy access to ports and devices from multiple angles.
- High Weight Capacity: Supports up to 300 lbs on the floor (200 lbs when adjusted to maximum depth) and 200 lbs when wall-mounted (depth cannot be adjusted in wall-mounted mode). Made from carbon steel for superior welding performance and durability, this open frame rack is designed to save space while accommodating multiple devices.
- User-Friendly Design: Designed with your convenience in mind, this open frame server rack features an top shelf for extra storage and improved space utilization. The rolling casters let you move it effortlessly wherever you need it, making setup and movement a breeze.
- Widely Applicable: Maximize your space with this adaptable open frame server rack, designed to make the most of every inch. Ideal for retail spots, classrooms, offices, and any area where space is at a premium, it delivers practical solutions for your storage needs.
- Everything You Need: Our open-frame rack comes with fully equipped accessory kit for easy setup and secure installation: 2 x Trays, 4 x Casters, 1 x set of Screws, 16 x M6*12 Cage Nuts, 1 x Grounding Wire, 1 x Internal & External Hex Wrenches, and 1 x User Manual.
Different defaults and allowlists
Cloudflare’s file is described as an affirmative allowlist for several AI crawler tokens, with a comment about access to markdown versions of pages. Netflix uses a different pattern: a wildcard disallow alongside an allowlist that includes named search and AI crawlers. The wildcard default is therefore central to interpreting what a named exception does.
Path-specific rules and purpose-specific tokens
Moz blocks GPTBot from /blog/ and /learn/seo/, rather than blocking it site-wide. eBay distinguishes GPTBot and Applebot-Extended from OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and Claude-User. LinkedIn likewise treats GPTBot, ChatGPT-User, and OAI-SearchBot differently.
Rank #4
- ADJUSTABLE DEPTH: 4-Post 42U open frame server rack with 4 vertical rails and adjustable mounting depth 22" to 40" (56,0cm to 101,7cm); Compatible with various servers / switches / data / AV and other IT equipment; EIA/ECA-310-E Compliant
- EASY ASSEMBLY: Mobile network rack with easy-to-follow assembly instructions and online video; Compact flat-pack shipping to avoid damage and facilitate installation; Total product height of 80.3in (204 cm) with casters, 78in (198cm) without casters
- COLD ROLLED STEEL: Durable 4 Post 19in open frame rack designed for ventilation with 42U mounting height and 1320lb (600kg) weight capacity (stationary); 3 install options included: casters, levelling feet, or base-plate to secure rack to the floor
- HARDWARE INCLUDED: Rolling computer/data rack includes cage nuts and screws to mount equipment, easy to read Units (U) and depth adjustment markings, cable management hooks for organization, and required assembly tools
- THE IT PRO'S CHOICE: Designed and built for IT Professionals, this 42U rack is backed for 2-years, including free lifetime 24/5 multi-lingual technical assistance
Shared groups can affect several crawlers
GitHub places several crawler tokens in a shared group with a crawl delay and a marketing-page allowlist, while giving Bytespider a separate block. When tokens share a group, changing a directive can affect all of them; inspect group membership before editing a rule.
Free tools Windows power users keep installed
One-click scans. No signup required.
How to check what a robots.txt policy means
- Identify the crawler token and its purpose. Do not assume that a training crawler, a search crawler, and a user-triggered retrieval crawler are interchangeable. The study’s examples show that sites can set different rules for each.
- Find the applicable group. Check for a group matching the crawler token. If there is no crawler-specific group, determine what the wildcard group says; silence in a named group is not enough to establish the effective rule.
- Match the actual URL path. Test more than the homepage: include an article URL and a URL you believe should be blocked. A path-specific rule may leave the root available, and the most specific matching rule matters.
- Read empty directives literally.
Disallow:with no path disallows nothing. Do not treat an empty value as equivalent toDisallow: /. - Use a robots.txt tester and review the result. A tester can help evaluate how rules match representative URLs, but it cannot establish whether a crawler follows the policy or whether another access control is in place.
What this snapshot cannot establish
- Prevalence across the web: the 70 domains were hand-picked, so their policies do not represent all sites.
- Rules for every crawler: the study covered 22 selected tokens; unexamined and newly introduced tokens are outside its results.
- Behavior over time: these files were fetched on one day, September 23, 2026, and may change.
- Actual access or compliance: a robots.txt file states a policy request; it does not prove that a crawler obeyed it or that a site has no other restrictions.
SerpPrism reported no misspelled AI crawler tokens in this sample. It also found no site blocking Googlebot except Reddit, whose wildcard rule blocks everyone covered by it. Those are observations about these 70 files, not proof that such cases never occur elsewhere.
Quick Recap
Best Value
- Standard 1U Height: Get more space with our 1U server rack shelf—it comes in a set of 4! Ideal for 19-inch 4-post server racks, stacking routers, switches, firewalls, and other network gear. Easy storage and a neat setup in one simple solution
- Heavy-Duty Construction: Crafted from premium Q235 carbon steel with a robust 0.06 in (1.5 mm) thickness, our network rack shelf can handle up to 50 lbs (22.68 kg) with ease. Say goodbye to wobbles and tilts—keeping everything in its place
- Optimal Ventilation: Featuring a vented bottom design, our rack mount shelf effectively reduces equipment temperature, ensuring stable operation and lowering the risk of malfunctions. Keep your gear running smoothly for longer-lasting performance
- Flexible Partitioning: Each shelf features a depth of 10 in (254 mm). Our server rack shelf helps you organize and optimize your rack space efficiently. Keep your equipment neatly separated to reduce clutter and minimize interference or collisions
- Installation Made Easy: Everything you need for installation is included—screws and nuts are provided, making the process quick and hassle-free. Simply use a Phillips screwdriver, and you'll have your network rack shelf installed in no time
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
