Free tools Windows power users keep installed
One-click scans. No signup required.
llms.txt is a proposed Markdown index that helps AI agents navigate a website, especially its documentation. Version 2 clarifies how files can be scoped to parts of a site and how pages can advertise Markdown versions, but it does not make llms.txt a crawler directive or establish it as a search-visibility tactic. In Ahrefs’ May 2026 sample of 137,210 traffic-active domains, 28% had a valid root file; 97% of those files received no requests during the month.
What is llms.txt?
The proposal, authored by Jeremy Howard, describes llms.txt as a compact Markdown guide to a website: a short introduction and curated links that help an agent find relevant material. It is especially suited to documentation, where an index can point to detailed pages in a format that is easier for tools to consume.
As an Amazon Associate I earn from qualifying purchases.
It is not equivalent to robots.txt. It does not tell crawlers what they may access, block them, or direct search indexing. Its intended role is navigation. The proposal was first published September 3, 2024; its current version 2 page was modified August 10, 2026: llmstxt.org specification.
Recommended Free Tools
What does version 2 specify?
File location and scope
A file may be placed at the site root or at a subpath. It covers URLs beneath its own path; if multiple files could apply, the most specific applicable file takes precedence. That means a documentation area can have its own index without requiring a single file to represent every section of a large site.
#1 Best Overall
Required and recommended content
The only required section is an H1 naming the site or project. The format describes a short summary in a blockquote, optional explanatory material, and H2 sections containing lists of links to detailed pages. The summary is recommended, not mandatory.
The structure is meant to make an index readable and useful, not to impose a complex schema. The important practical choice is to curate links that help an agent reach authoritative, relevant pages rather than dumping an unmaintained URL list.
Markdown page discovery
Version 2 describes two patterns for a Markdown version of an HTML page: append .md to the original path, such as page.html.md, or replace its extension, such as page.md. A page can advertise its Markdown version with rel="alternate" type="text/markdown"; it can also identify the applicable llms.txt using rel="describedby". Either relation can be delivered in an HTML <link> element or an HTTP Link: header. See the version 2 change notes.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Rank #2
What “Optional” means
An H2 titled Optional remains a convention for secondary links an agent might skip when it needs less context. Version 2 removed the earlier context-expansion mechanism that gave the heading mechanical omission semantics. A parser is not required by the current proposal to omit everything under that heading.
What is not part of the v2 specification
llms-full.txt is an ecosystem convention, not a file defined by the current v2 proposal. The older context-expansion companion-file mechanism is no longer part of the current proposal either.
What did the 137,000-domain study find?
Ahrefs published its analysis on June 15, 2026, using domains in Ahrefs Web Analytics that received traffic in May 2026. It checked each root for an HTTP 200 response at llms.txt, screened out HTML and soft-error pages, and classified requests to those paths with Ahrefs Bot Analytics.
Here are the main results, with their denominators kept distinct:
| Finding | What it measures |
|---|---|
28% of 137,210 domains had a valid root llms.txt. |
Share of the Ahrefs traffic-active domain panel, not the whole web. |
| 97% of approximately 38,000 valid files received zero requests. | Share of valid files during May 2026. |
| 96% of requests to files that received traffic came from bots. | Share of requests to files with activity. |
| 19.5% of requests to files that received traffic came from named AI-tool categories. | Share of active-file requests; categories include AI agents, training crawlers, assistants, and retrieval bots. |
| 1.1% of measured requests came from AI retrieval bots. | Share of active-file requests attributed to live-query retrieval crawlers. |
12% of measured requests came from tools studying llms.txt. |
Share of active-file requests attributed to SEO/GEO/AEO auditing, discovery, checking, and research tools. |
These figures come from Ahrefs’ study by Louise Linehan. Ahrefs says its customers skew more technical and SEO-aware than the web overall, so the 28% result is a panel-specific figure, not a representative estimate of adoption across all websites. The study also did not assess whether files conformed to the specification.
The request percentages describe traffic to files that received requests; they do not mean that the same share of all published files was read. In particular, 19.5% is not the share of published files consulted by AI, and 1.1% is a narrow retrieval-bot category rather than a measure of every AI-related request. The study captures one month in one vendor’s panel. It cannot establish future adoption, universal use, or a causal effect on search visibility.
Does llms.txt improve AI search visibility?
The available evidence does not establish llms.txt as a ranking or AI-citation lever. Ahrefs reports that Google’s generative-AI guidance says machine-readable files such as llms.txt are not needed to appear in generative AI search. It also reports that Google Search Advocate John Mueller described the file as “not done for search,” while suggesting it could temporarily save tokens for AI coding tools parsing developer documentation. That wording is reported by Ahrefs; it should not be read as proof that no agent will ever use the format.
The narrower, supportable conclusion is that llms.txt is a proposed documentation navigation aid, not an established general-purpose search visibility tactic. Ahrefs’ request data does show some bot and AI-tool activity, but most valid files in its sample received no requests during the measured month.
Should you publish one?
For a documentation-heavy site, a concise, maintained file can be a modest navigation improvement, particularly if the team already publishes Markdown documentation. For a site focused primarily on search visibility, the evidence here does not justify prioritizing it as a proven ranking or citation strategy.
Best Value
- Core i3-8100 3.6GHz 4-Core Processor
- 8GB (1x 8GB) DDR4 Memory
- 480GB SATA 6Gbps SSD
- 2x Integrated 1GbE RJ45 NIC Ports -- Bring Your Own PCIe NIC
- Shallow/Short Depth Rackmount Server
Decide based on the actual documentation and maintenance trade-off, rather than a promise of SEO impact:
- Useful content to navigate: Do agents need help locating substantial, structured documentation or other authoritative pages?
- Maintenance approach: Can your team keep curated links current, or generate the index alongside existing documentation without letting it become stale?
- Markdown availability: Are machine-readable page versions already served, or would supporting them add meaningful work?
- Discovery relations: Can you expose and maintain the proposed alternate and described-by relations accurately?
- Opportunity cost: Is maintaining the file more valuable than other documentation work your team could do?
If you publish one, keep it short, useful, and aligned with the site’s real structure. Point to high-value machine-readable pages where available, and validate that declared Markdown URLs work. A documentation platform such as Mintlify or GitBook may be relevant if you are already choosing documentation infrastructure; its use is not evidence that buying it improves AI visibility.
Quick Recap
Common misconceptions about llms.txt
- “It must be at the root.” Version 2 allows files at subpaths and defines how their scopes apply.
- “The summary blockquote is compulsory.” The H1 is the only required section; a summary is recommended.
- “Optional tells every parser what to skip.” It is a convention in v2, not a mechanical omission instruction.
- “llms-full.txt is part of the standard.” It is an ecosystem convention, not defined by the current v2 proposal.
- “28% of the web has the file.” The figure describes Ahrefs’ May 2026 traffic-active panel, which Ahrefs says is more technical and SEO-aware than the web overall.
- “The W3C has a Working Draft for it.” A cross-check published September 20, 2026, describes the cited W3C item as an open repository issue, not a W3C Working Draft: Ievgenii Gryshkun’s comparison of v2 and the study.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitches
