Yes, at least some AI crawlers have requested sitemap and RSS files: Google Search Relations lead John Mueller said he had seen those requests in his server logs. That shows the files can be discovered and fetched—not that every listed page was indexed, used for AI training, or cited in an AI answer. The crawlers were not identified, and their use of the files was not established.
What Mueller reported about AI crawlers
In a report published October 5, 2026, Search Engine Journal described comments by Mueller and Martin Splitt on the October 1 episode of Google’s Search Off the Record podcast, “Do sitemaps still matter?” Mueller said he had seen an AI crawler request his sitemap in server logs and had also seen similar requests for RSS files. He did not name the crawlers or know whether AI companies documented the behavior or what they did with the files. Search Engine Journal’s report
As an Amazon Associate I earn from qualifying purchases.
Mueller’s practical point was about discovery. He said AI training crawlers usually do not provide a console or other setup for submitting a sitemap. For publishers who want such crawlers to find their content, he suggested using a conventional sitemap name such as sitemap.xml or offering RSS feeds. Feeds are often discoverable through links in a page’s HTML head.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsWhat a sitemap or feed request does—and does not—tell you
A server-log entry records a request to a file. It does not show that the requesting system successfully processed the file, fetched every URL listed in it, indexed those pages, trained on their content, or used them in an AI-generated answer. Those are separate steps, and Mueller’s report did not establish that any occurred.
#1 Best Overall
The distinction also matters for Google. Google Search Central says sitemap submission is a hint, not a guarantee that Google will download the sitemap or crawl its listed URLs. A request for a sitemap is therefore evidence of access, not proof of downstream use.
Sitemap or RSS/Atom feed: which should you provide?
| Consideration | Sitemap | RSS or Atom feed |
|---|---|---|
| Coverage | Can list a broader set of site URLs. Google’s documented limit is 50 MB uncompressed or 50,000 URLs per sitemap; larger sets can be split across files and organized with a sitemap index. | Usually covers recent URLs, so it is not a complete inventory of an older or larger site. |
| Discovery | A conventional filename such as sitemap.xml and an absolute sitemap URL in robots.txt can make it easier to find. A Sitemap entry in robots.txt is separate from the user-agent-specific rules. |
Often discoverable through feed links in the HTML head; many content-management systems generate feeds automatically. |
| Format and detail | Google supports the formats defined by the Sitemaps protocol and says it has no preference among supported formats. XML is the most versatile and can carry extra information for images, video, news, and localized pages. | RSS, mRSS, and Atom can be convenient, but have limitations compared with XML sitemaps and generally include only recent URLs. |
| Maintenance | Many content-management systems generate sitemaps automatically; check what your site already publishes before adding a tool. | Many content-management systems also generate feeds automatically; check existing feed links before setting up another one. |
| Guarantee of crawling or use | None. A sitemap is a discovery hint, not a guarantee that a crawler will fetch or use it. | None. A feed can expose recent URLs, but that does not guarantee a crawler will fetch or use them. |
Google recommends absolute URLs in sitemaps. A sitemap in the site root can cover all files on that site; without Search Console submission, a sitemap affects only descendants of its parent directory. Google says it may use an accurate lastmod value when it can verify it consistently, but ignores priority and changefreq. These are Google’s documented behaviors and should not be assumed to describe every AI crawler. Google Search Central: Build and submit a sitemap
How to make sitemap and feed files discoverable
- Check what your site already publishes. Look in your CMS or site configuration for a sitemap and RSS or Atom feed before installing another plugin or service. Google notes that most CMSes generate sitemaps or feeds automatically.
- Use a conventional sitemap location. If you publish a sitemap, a standard name such as
sitemap.xmlis the naming approach Mueller suggested for crawlers that lack a submission console. - Consider listing its absolute URL in
robots.txt. Google documents this as one way to make a sitemap available. The Sitemap line is not tied to one user-agent’s rules, but listing it does not guarantee that a crawler will use it. - Expose feeds clearly. If you publish an RSS or Atom feed, make it discoverable from the site; feeds are often linked from the HTML head. Remember that feeds generally cover recent URLs rather than every page.
- Inspect access logs for requests. A request can help you see that a client reached a file. Treat the user-agent string and request as observations to investigate, not conclusive proof of the bot’s identity or what it did with the contents.
What to do if Search Console says “Couldn’t fetch”
Do not assume the sitemap XML is malformed based on that message alone. Mueller’s reported explanation included host load and crawl demand as possible reasons a sitemap could not be fetched. Check whether the sitemap URL is reachable and whether the host was under strain when the request occurred, then review the sitemap’s status in Search Console. Google’s own guidance remains that submitting a sitemap does not guarantee it will be downloaded or that its URLs will be crawled. Google Search Central: Build and submit a sitemap · Search Engine Journal’s report on Mueller’s comments
Recommended Free Tools
What Mueller said about llms.txt
In the Search Engine Journal report, Mueller compared llms.txt with an HTML sitemap and said it does not meet the strict format Google requires for a sitemap. He reportedly characterized support in the systems discussed as lacking and advised against relying on it. This is not evidence that every AI crawler ignores llms.txt; the report does not establish how all such crawlers handle the file.
Quick Recap
Rank #3
- Keep track of everything from attendance to test scores
- Spiral bound
- Measures 8-1/2" x 11"
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
