In a hand-picked snapshot fetched on September 23, 2026, 54 of 71 usable robots.txt files declared at least one sitemap, 26 declared more than one, and 17 had no Sitemap line. These figures describe only the selected sites on that date—not all websites. The findings are useful for seeing how sites configure the file, but Google’s documentation is what determines what Googlebot supports.
What the 71-file snapshot found
SerpPrism’s survey, by Hongtao Ren, fetched /robots.txt from 78 manually chosen high-traffic domains across news, ecommerce, SaaS, developer, social, finance, education, and government categories. Seventy-one responses were usable plain-text files: 46 arrived directly and 25 through a local proxy. Two responses were HTML, while five could not be read—four returned HTTP 403 or 418 and one returned 404. The author notes that direct and proxy retrieval was not evenly distributed across categories, so this is a one-day snapshot of a selected set, not a representative estimate of the web. SerpPrism’s article and published domain list describe the method and results.
As an Amazon Associate I earn from qualifying purchases.
| Observation | Result in the 71 usable files |
|---|---|
| At least one Sitemap line | 54 of 71 (76%) |
| More than one Sitemap line | 26 of 71 |
| No Sitemap line | 17 of 71 (24%) |
Crawl-delay in the wildcard group |
5 of 71 |
| A declared sitemap blocked by the same file’s Disallow rule | 0 of 71 |
These are counts of what the survey’s line-by-line parser found, respecting user-agent groups. They are not rates for all sites, and individual files may have changed since the fetch date.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Sites with no Sitemap line
The survey named Forbes, Amazon, Etsy, Shopify, GitHub, GitLab, Reddit, LinkedIn, Quora, npmjs.com, Python.org, Go.dev, Mozilla.org, W3.org, Ahrefs, Screaming Frog, and MIT among the 17 files without a Sitemap line. That observation means only that the inspected response did not declare a sitemap there. It does not establish that those sites lacked sitemaps.
Is a sitemap in robots.txt required?
No. A Sitemap line is one discovery route, not a requirement. Google says site owners can also submit a sitemap through Search Console; submission is a hint, not a promise that Google will fetch or use it. So a missing line in robots.txt is not evidence that a site has no sitemap or that Google cannot discover one. Google’s sitemap guidance covers the available submission options.
Does Google support Crawl-delay?
No. Five files in the snapshot placed Crawl-delay in their User-agent: * groups. The survey also reports that GitHub had a separate AI-crawler group naming GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, and PerplexityBot, with Crawl-delay: 1. These are observations of those files as fetched on September 23, 2026; they do not establish their current contents or how every named crawler interprets them.
For Google specifically, the answer is unambiguous: its specification lists supported fields and says other fields, including crawl-delay, are not supported. Do not use that directive to control Googlebot. Google’s robots.txt specification documents the supported fields: user-agent, allow, disallow, and sitemap.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteCan a sitemap live on another domain?
Yes. Google’s documentation says a Sitemap URL in robots.txt must be fully qualified, but it does not have to use the same host as the robots.txt file; the field may also appear multiple times. In the survey, notion.so listed 11 sitemap URLs on www.notion.com, and trello.com listed one on a594014.sitemaphosting7.com. Those are valid cross-host patterns under Google’s documented guidance, not protocol violations.
A cross-host declaration does create an operational dependency: the sitemap host needs to remain reachable and serve the intended file. That is a maintenance consideration, separate from whether Google permits the arrangement. The survey also found HTTP sitemap URLs in theguardian.com and who.int files; both reportedly redirected. That dated observation is a reason to check current destinations, not proof the declarations failed.
What should I check in my own robots.txt?
Google describes robots.txt as a UTF-8 plain-text file at the top level for the applicable host, protocol, and port. A file for one host or protocol does not automatically govern another. When checking your setup, inspect both the response and the rules that apply to the crawler in question.
- Request the exact file. Check the relevant origin’s
/robots.txt, including its protocol, host, and port. Inspect the response body rather than relying only on the status code. - Check the format and content. Google expects UTF-8 plain text. In this snapshot, khanacademy.org and cdc.gov returned HTTP 200 with HTML; SerpPrism characterized those responses as soft 404s. Google says it attempts to parse HTML responses for rules and ignores other content, so validate what the body actually contains rather than assuming every crawler treats HTML as total failure.
- Read the right user-agent group. Confirm that each
AlloworDisallowrule is in the group for the crawler you care about. A crawler-specific group is not automatically a universal rule. - Check sitemap discovery separately. Look for Sitemap lines, but if none appear, check Search Console and other discovery routes before concluding no sitemap exists.
- Use directives for the right goal. Use supported crawl rules for crawl access; do not expect
Crawl-delayto control Google. For index exclusion or privacy, use the appropriate controls instead of treating robots.txt as a security boundary.
Among the 71 usable files in this survey, none declared a sitemap and then blocked that sitemap with its own Disallow rule. That is a sample result, not a guarantee that such a configuration never occurs.
Does robots.txt keep a page out of Google?
No. Robots.txt manages crawler access; it is not a dependable way to remove a URL from Google’s index. Google may still show a blocked URL without crawling its contents. If a page should not appear in Search, Google advises allowing it to be crawled so a noindex directive can be seen. If the content is private, protect it with authentication or another access-control mechanism; a robots.txt rule does not keep people or all crawlers out. Google’s robots.txt introduction explains the distinction between crawl control, indexing, and access protection.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
