What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Short answer: llms.txt has not been shown to improve Google rankings, AI Overviews visibility, or the rate at which AI systems cite a site. In Ahrefs’ May 2026 server-log sample, 97% of valid files received no requests at all. Other analyses likewise found little fetching and no measurable citation advantage. A small, automatically generated file can still be a reasonable index for coding agents working through technical documentation, but that is a narrow operational benefit—not an SEO signal.
That distinction matters. Publishing llms.txt will not compensate for blocked crawling, weak page structure, or outdated documentation. Treat it as optional documentation plumbing, measure whether any useful agent actually requests it, and keep your main effort on pages that search engines and people can already access.
As an Amazon Associate I earn from qualifying purchases.
Recommended Free Tools
What is llms.txt, and what is it supposed to do?
llms.txt is a proposed Markdown file placed at a site’s root, normally at /llms.txt. The proposal, introduced by Jeremy Howard in September 2024, asks site owners to provide a short description of the site and links to selected, authoritative pages. An AI system could read that index instead of discovering every page independently.
It is a community proposal, not a standards-body protocol. It has no enforcement mechanism and does not tell a crawler what it may or may not fetch. That is the key difference from robots.txt: robots.txt communicates crawler access rules, while llms.txt is merely information a willing client may choose to retrieve.
Popular on Amazon
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
The proposed file therefore cannot guarantee that a model will read your pages, use the file in an answer, or cite your domain. A request in a server log only proves that a client fetched the file; it does not prove that the contents influenced a response.
Does llms.txt improve AI-search visibility?
No measured study cited here shows an increase in rankings, AI-answer inclusion, traffic, or citation rate after a site publishes llms.txt. The available numbers mostly describe adoption and requests, and the strongest samples show that actual retrieval is uncommon.
| Source and scope | Observed result | What it means |
|---|---|---|
| Ahrefs, 137,210 domains; May 2026 logs | 97% of valid llms.txt files received zero requests. About 28% of domains had a valid file. | Most files were not fetched. The 28% figure comes from an analytics-customer sample that is more technical and SEO-aware than the wider web, so Ahrefs describes it as an upper bound for adoption. |
| Ahrefs request breakdown, same sample | Of the requests that did occur, 19.5% came from named AI tools and 12% from GEO/AEO or llms.txt-checking tools. | Even a request is not proof that a model used the file or that a site gained visibility. |
| OtterlyAI, 90-day test summarized by Optimizely | 84 of more than 62,100 AI-bot visits touched llms.txt, approximately 0.1%. | In that test, bots almost always followed other discovery paths. |
| SE Ranking, roughly 300,000 domains, 2025 | 10.13% adoption and no clear relationship between having llms.txt and receiving AI citations. | Having the file was not associated with a citation advantage in that analysis. |
| llms.txt Studio benchmark, 20 domains, five questions, four retrieval routes and 400 attempts, 2026 | Files below roughly 10,000 tokens were about twice as cheap to read; files above that size were about nine times more expensive. | Large files can consume more model context than they save. The authors warn that this small sample is not generalizable. |
These results are consistent: publishing is easy, but routine retrieval is rare, and no cited analysis demonstrates a search or citation lift. Correlation studies cannot establish causation, and a fetch count cannot tell you what happened inside a model after retrieval.
Does Google use llms.txt for Search, AI Overviews, or AI Mode?
The reviewed Google guidance says machine-readable files such as llms.txt are not needed for Google Search or AI Overviews, and the same evidence reports no use for AI Mode. Google’s search systems continue to evaluate crawlable pages, their content, links, structure and other established signals rather than treating llms.txt as a ranking requirement.
Do not confuse a Google product with a Google development tool. Chrome’s Lighthouse team has added an experimental Agentic Browsing audit that checks whether a site has an llms.txt file. That audit can encourage adoption or expose a missing file during a test, but it is not evidence that Google Search uses the file or that passing the audit improves visibility.
Also popular on Amazon
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Google’s John Mueller was quoted by Ahrefs describing llms.txt as “not done for search” and as a “temporary crutch, perhaps to save some tokens” for AI coding tools parsing developer documentation. That is a useful description of its current role: convenience for a narrow client, not a search directive.
Do AI crawlers read llms.txt?
Some clients appear to retrieve it, but the provider statements and logs do not establish a business outcome. Anthropic and Perplexity have said they read or retrieve llms.txt, while OpenAI’s position is described as uncommitted in the reviewed material.
Ahrefs’ logs identify Claude Code among the more frequent named fetchers in the small set of requests that reached llms.txt. That indicates a practical path for some coding workflows, not a guarantee that a general-purpose chatbot will consult the file when answering a user.
Bot names also need careful interpretation. A named AI crawler, a documentation assistant, a monitoring service and a GEO/AEO checker can all request the same URL. Separate them in your logs before claiming that “AI traffic” read the file, and remember that user-agent labels can be incomplete or spoofed.
Is llms.txt worth publishing for SEO?
For general search marketing, do not publish it expecting a visibility lift. For a documentation-heavy site, a compact, generated index may be worth the small engineering cost if it helps a coding agent find canonical references. Choose based on the consumer you want to serve, not on the existence of a trend.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteAlso popular on Amazon
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
| Decision factor | Publish a small, generated file | Skip it for now |
|---|---|---|
| Target consumer | Documentation and API agents that need a map of canonical pages. | Marketing teams seeking higher rankings, AI Overviews placement or citations. |
| Evidence of fetching | Your logs show identifiable, useful agent requests, such as documentation tools. | No requests, or only scanner and checker traffic. |
| Maintenance | Generated from the same registry as your sitemap or documentation index, so links update with releases. | Maintained by hand, with a high risk of stale links and obsolete descriptions. |
| File size | Short enough to summarize the site and point to the best pages. | Long enough to duplicate your entire documentation set or consume substantial model context. |
| Site fundamentals | Crawl access, canonical URLs and page structure are already healthy. | Important pages are blocked, thin, duplicated, slow or difficult to navigate. |
The practical summary from Optimizely is apt: “An llms.txt file tells AI what your site claims to be about. AI search would rather check.” Your pages still have to support the claims in the file.
When can llms.txt help documentation and coding agents?
Technical documentation and API-reference sites have the clearest rationale. A short index can identify the product, list the canonical getting-started and reference sections, and direct an agent away from archived versions, changelog noise or duplicate navigation pages.
That benefit is about discoverability inside a tool that chooses to read the file. It does not prove that the agent’s generated code will be correct, that users will see more of your pages, or that your organization will receive more leads. Validate those outcomes separately.
Use one source of truth
Generate llms.txt from the same content registry that produces your sitemap or documentation navigation. A release that renames or removes a page should update the file in the same deployment. This avoids the common failure where an attractive index points to retired versions.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Keep the index selective
Include the site’s purpose and a small set of canonical links rather than copying every paragraph of documentation. The benchmark’s token-cost result gives a practical warning: once a file grows beyond roughly 10,000 tokens, the context cost can rise sharply. The exact threshold will vary by model and retrieval route, so treat it as a design limit, not a universal law.
Also popular on Amazon
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Describe pages honestly
Use labels that match the destination: reference, installation, authentication, tutorials or support policy. Do not claim that a page is current, complete or authoritative if it is an experiment, an old version or a marketing summary.
How do you publish llms.txt without creating a maintenance problem?
Publishing is technically simple; keeping the file accurate is the real work. Use this procedure for a controlled rollout.
- Define the purpose. Decide whether the audience is a coding or documentation agent. If the only goal is SEO visibility, the available evidence does not justify the maintenance cost.
- Inventory canonical pages. Select the documentation, API references and policy pages that a new user or agent genuinely needs. Remove duplicates, redirects and retired versions.
- Generate from your existing registry. Build the Markdown output from the same source that drives your sitemap or navigation. Add the generation step to the normal documentation release process.
- Serve it at the root. Make the file available at
/llms.txtover the same HTTPS host as the pages it describes. Return a normal successful response and ensure your deployment does not accidentally send an HTML error page or require an interactive login. - Check every destination. Test that links resolve, canonical pages are crawlable, language or version choices are explicit, and descriptions still match the content after each release.
- Measure requests in your logs. Record requests for the file separately from ordinary page hits. Distinguish named AI tools, coding agents, GEO/AEO checkers, uptime monitors and unknown clients before interpreting the numbers.
- Review its value. If months of logs show no useful fetches, or only automated audits, remove the file or reduce it to a minimal index. If a documentation workflow demonstrably uses it, keep it generated and concise.
Does llms-full.txt work better?
There is no cited evidence here that an “llms-full.txt” variant improves ranking, AI citations or answer inclusion. A full-text file is, by definition, larger and therefore carries the same or greater context and staleness risks measured in the small token-cost benchmark.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →If an agent needs complete content, it can usually retrieve the linked canonical page. An index that points to well-maintained pages is easier to update than a second file that duplicates the entire documentation set. Use a larger file only when you have a specific client requirement and can measure that client’s benefit.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What does the current evidence not prove?
It does not prove that no model will ever use llms.txt. The findings show limited fetching and no demonstrated visibility effect in the cited samples; they do not predict every future provider or private enterprise agent.
Also popular on Amazon
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
It also does not prove that a request changed an answer. Ahrefs’ customer base is unusually technical and SEO-aware, SE Ranking’s analysis is observational, and the llms.txt Studio benchmark covers only 20 domains and 400 attempts. Those limits should make you cautious about both optimistic success stories and sweeping claims that every agent ignores the file.
What can be said plainly is narrower and more useful: no cited study demonstrates that publishing llms.txt increases rankings, traffic, answer inclusion or citation rate. Do not invent a success percentage to fill that gap.
What should you prioritize instead?
Make the pages you want cited easy for every legitimate crawler and reader to understand. Check access controls, robots directives, status codes, canonical URLs, internal links, structured headings, version labels and the freshness of technical examples. Then improve the substance of the answer those pages provide.
- Fix blocked or accidentally noindexed documentation before adding another discovery file.
- Consolidate duplicate or obsolete versions so an agent has one canonical destination.
- Write clear, self-contained pages that answer real questions and expose important facts in text.
- Keep API examples, authentication instructions and compatibility notes current.
- Use server logs and analytics to identify which clients actually request content, rather than assuming a user-agent name proves influence.
Once those foundations are sound, a small generated llms.txt file is a low-risk experiment for documentation sites. Treat it as optional navigation, set a review date, and keep it only if logs or user feedback show a concrete benefit.
Frequently Asked Questions
Can llms.txt stop an AI company from scraping my site?
No. The file is informational and does not enforce access rules. Use robots directives, authentication, rate limits and contractual controls for access decisions, while recognizing that each mechanism has different technical and legal limits.
How often should a documentation site update its llms.txt file?
Update it whenever the canonical documentation registry changes, ideally in the same deployment that publishes the page change. Automated generation is safer than a calendar reminder because it catches renamed and retired URLs immediately.
How can I tell whether a useful agent fetched my file?
Filter web-server logs for the exact path, response status, user-agent and source network, then separate coding assistants from scanners and SEO checkers. Compare those requests with subsequent visits to the linked documentation, but do not treat that sequence as proof that a model used the text in an answer.
Will a shorter file always be better?
Not automatically. It should contain enough context to identify the site and its canonical sections, but every extra passage adds staleness and context cost; remove anything that duplicates the linked pages.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
