Is llms.txt worth it? The 2026 evidence

Is llms.txt worth it?

llms.txt is a Markdown file at a website’s root that lists the pages a language model should read. As of September 2026, it is not worth adding to improve search visibility: Google says it neither helps nor harms rankings, and Ahrefs found that 97% of the files it studied received no requests at all. It is worth adding when coding agents or AI tools read your documentation.

That is a narrower answer than most guides give. The file is cheap to publish, so the real cost is not the file itself. The cost is treating it as a visibility tactic and skipping the work that search engines and answer engines do read: crawlable pages, clear answers and original evidence.

Is llms.txt actually used?

Rarely, and mostly not by the AI search bots people hope for. Ahrefs analyzed the server logs of about 137,000 domains and reported, on June 15, 2026, that 97% of llms.txt files received zero requests that month.

Among the files that were fetched, AI retrieval bots made about 1.1% of the requests. SEO audit tools made 21.7%, more than any other group. Ahrefs also found that Claude Code, Anthropic’s coding agent, fetched the files more often than any AI retrieval bot, AI assistant or AI training crawler except GPTBot and statespace-indexer, and concluded that AI tools fetch llms.txt when a link, an index or a user instruction tells them it exists.

A crawl log entry is also weaker evidence than it looks. In August 2026, Mark Williams-Cook published cats.txt, an invented standard for declaring office cats. AI crawlers fetched it, Google indexed it, and language models repeated details from it. His point was that any text file on the open web gets the same treatment, so those four signals prove nothing about llms.txt either.

Is llms.txt a standard?

No. It is a proposal. Jeremy Howard of Answer.AI published it on September 3, 2024, and the specification lives at llmstxt.org. Its authors expected it to be useful mainly at inference time, meaning the moment a model assembles context to answer a question, rather than for training.

The format is deliberately small. The only required element is an H1 with the site or project name. A blockquote summary, H2 sections listing links, and an “Optional” section for material a model can skip are recommended. The proposal also suggests offering a clean Markdown copy of each page at the same URL with .md appended.

llms.txt vs robots.txt: what’s the difference?

They do different jobs and have different standing. On purpose, robots.txt tells crawlers what they may access, while llms.txt suggests what a model should read. On status, robots.txt became an IETF standard, RFC 9309, in September 2022, while llms.txt remains a proposal.

On readers, robots.txt is honored by the major search crawlers, while the largest share of llms.txt requests came from SEO audit tools, according to the Ahrefs data. On effect in Google Search, a robots.txt rule can keep a page out of the crawl, while Google says an llms.txt file will neither help nor harm a site’s visibility. Adding one never replaces the other.

Does llms.txt help SEO?

Not in Google. Google’s guide to its generative AI features, last updated July 10, 2026, says site owners don’t need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, including AI Overviews and AI Mode.

The same guide says it is fine to maintain an llms.txt file for other services that use it, and that doing so will neither harm nor help visibility in Google Search. It treats structured data the same way: not required for generative AI features, but still worth keeping because it supports rich results. For other answer engines, we found no public commitment to use the file as a ranking or citation signal, so we don’t claim one.

What we did on phasekind.com

We publish both /llms.txt and /llms-full.txt, and neither is written by hand. They are generated at build time from the same data as the site’s pages: the services, the case studies and these notes. When a note is published, it appears in both files and in the sitemap from the same entry, so the file cannot drift from the pages it describes.

We have not measured who fetches our file, so we make no claim that it has earned a citation. The method we would use is simple: count a month of requests to /llms.txt by user agent, separate AI retrieval bots from audit tools and coding agents, and compare that with requests to the pages themselves. Until a check like that shows otherwise, we treat the file as a courtesy to tools that ask for it, and put the effort into the pages.