Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesrobots.txt tells crawlers which paths they are asked to access or avoid; sitemap.xml helps search engines discover URLs; and llms.txt is a proposal for a concise, curated overview intended for LLM agents that choose to use it. They serve different purposes and none replaces the others. In particular, robots.txt is not security, a sitemap cannot guarantee indexing, and Google says llms.txt does not affect Google Search visibility or rankings.
How the three files differ
| File | Primary job | Intended consumer | What it contains | Status and limit |
|---|---|---|---|---|
robots.txt |
Communicates crawler access preferences for URL paths | Automated crawlers | User-agent groups and allow/disallow rules; crawlers may also recognize a sitemap location | The Robots Exclusion Protocol is specified in IETF RFC 9309. Its rules are not access authorization. |
sitemap.xml |
Helps search systems discover site URLs and associated information | Search engines | URLs and optional metadata or extensions, such as last-modified dates, images, video, news, and language alternatives | Supported sitemap formats help discovery, but inclusion does not guarantee crawling or indexing. |
llms.txt |
Proposes a curated, LLM-readable site overview | LLM agents or tools that choose to use it | A brief site overview and selected links or material | It is a proposal, not a replacement for either crawler instructions or a search sitemap. Google says it has no effect on Google Search visibility or rankings. |
What robots.txt does—and does not do
robots.txt is a convention for communicating crawler preferences, not a lock on content. RFC 9309, the IETF’s September 2022 Robots Exclusion Protocol specification, says its rules are not a form of access authorization. A crawler may honor a disallow rule, but that does not make a private file private.
As an Amazon Associate I earn from qualifying purchases.
Use application- or server-level authentication and authorization to protect restricted material. Do not rely on Disallow to hide it: the paths named in a robots file are public and discoverable.
Where it goes and what it contains
Under RFC 9309, the file is named /robots.txt (lowercase) at the service’s top-level path. It must use UTF-8 and the text/plain media type. Rules are organized into crawler user-agent groups and allow/disallow directives.
The file can also contain a Sitemap: record, for example Sitemap: https://example.com/my_sitemap.xml. RFC 9309 classifies that as an “other record,” outside the protocol’s core rules; Google documents this form and says it will find the sitemap the next time it crawls robots.txt. Monitor the file’s availability and test changes carefully: RFC 9309 specifies that crawlers encountering server or network errors when retrieving it assume complete disallow under the protocol’s error behavior.
Read IETF RFC 9309 for the protocol’s requirements and crawler behavior.
What sitemap.xml does—and when you need one
A sitemap gives search engines a list of URLs a publisher wants them to discover, with optional supporting information. Google describes a sitemap as a discovery aid, not a submission that compels crawling or indexing: it may attempt to crawl listed URLs, but inclusion does not guarantee that a URL will be crawled or appear in results.
Free tools Windows power users keep installed
One-click scans. No signup required.
When it is most useful
- Large sites, where search engines may benefit from a structured URL list.
- New sites with few external links pointing to them.
- Sites with rich media or news content that can use relevant sitemap extensions.
Google says a sitemap may be unnecessary for a small site of about 500 relevant pages or fewer if the site is comprehensively internally linked and has few relevant media or news files. That is conditional guidance, not a universal page-count cutoff.
Build a sitemap that helps rather than misleads
- List canonical URLs you want considered for Google Search, using fully qualified absolute URLs and UTF-8.
- Google’s current guidance permits up to 50 MB uncompressed or 50,000 URLs in one sitemap. Larger sites can split sitemaps and use a sitemap index.
- Use
<lastmod>only when it consistently and verifiably reflects a significant change to the page, such as content, structured data, or links—not a routine copyright-year update. - Do not spend effort tuning
<priority>or<changefreq>for Google: it ignores both.
XML is the most versatile sitemap format supported by Google. Text sitemaps are simpler, but list only URLs for HTML and other indexable text. Google’s guidance is at Build and Submit a Sitemap and its sitemap overview.
What llms.txt is intended to do
The llms.txt proposal describes a standard-named file at a site’s root containing a concise overview and selected links or material for LLM agents to consult on demand. It is meant to offer curated context, not to enumerate every URL or tell crawlers which paths they may access. It is also not established that every AI product reads or uses the file.
The proposal argues that a sitemap may be too large for an LLM context, may not point to LLM-readable versions, and may omit useful external material. Those are the proposal’s rationale for a separate overview; they should not be taken as a universal description of how every model or service handles sitemaps.
Google’s June 15, 2026 documentation update says llms.txt files are not needed for Google Search and will not positively or negatively affect visibility or rankings. A site may still maintain one for other services that use it, but adding the file is not a basis for expecting Google traffic, AI citations, or ranking changes. See the Google Search documentation updates and the llms.txt proposal.
Which files should you publish?
- Use robots.txt to communicate crawler access preferences. Use real access controls for anything confidential or restricted.
- Use a sitemap when it will help expose URLs, especially on a large, new, or media-rich site. Keep its URLs and metadata accurate.
- Consider llms.txt only if you want to provide a curated overview for AI tools that choose to consume it. It is optional for Google Search.
A site can publish all three: their jobs are complementary. A sitemap location in robots.txt can help crawlers find the sitemap, while llms.txt offers a separate proposed format for curated LLM context.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
