llms.txt Explained: What It Does, Who Actually Reads It, and Whether You Need One

Almost every article about llms.txt tells you to add one immediately. Very few tell you that Google has publicly said you can ignore it, or that no independent study has yet shown it changes how often you get cited.

Both things can be true at once: llms.txt is cheap, harmless and genuinely useful for a specific set of sites, and it is also being oversold as an AI ranking lever that it is not.

This guide sets out what the file actually is, what the major AI companies have said about it on the record, what the evidence shows, and how to decide whether writing one is worth an hour of your time.

What llms.txt is

llms.txt is a plain Markdown file published at the root of your domain, at /llms.txt. It was proposed by Jeremy Howard of Answer.AI in September 2024.

The problem it tries to solve is straightforward. When a large language model needs information from your website, it has to work with a limited context window. A modern HTML page is mostly not content: it is navigation, cookie banners, scripts, tracking, sidebars, related-post modules and footers. Feeding that to a model wastes most of the available context on markup the model has to discard.

llms.txt offers a shortcut. Rather than making the model reconstruct your site from HTML, you hand it a curated, human-written map: here is what this site is, here are the pages that matter, here is what each one covers.

The proposal assumes a model that is short on context and long on markup to discard.

1. 01Model needs an answerAn assistant is asked something your site can answer well.
2. 02It requests the mapInstead of crawling blind, it fetches /llms.txt.
3. 03It picks the right pagesYour one-line descriptions do the routing work.
4. 04It reads clean contentMarkdown, not navigation and cookie banners.

That is the theory. Whether it happens in practice is the part most articles skip.

What llms.txt is not

The single most common misunderstanding is that llms.txt is the AI equivalent of robots.txt. It is not, and the difference matters.

robots.txt

A permissions file. It tells crawlers what they may and may not fetch. It is widely honoured, decades old, and it is where you actually control AI crawler access, using agents such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended.

llms.txt

A recommendations file. It suggests what is worth reading and gives each page a description. It grants no permissions, blocks nothing, and has no enforcement behind it. Publishing one does not stop a single crawler doing anything.

Three further things llms.txt is not:

It is not a ranking factor. There is no algorithm that reads it and adjusts a position. Google has said so directly.

It is not a sitemap. A sitemap is machine-generated, exhaustive and tells crawlers what exists. llms.txt is hand-written, deliberately selective and tells readers what matters. Publishing every URL defeats the point.

It is not a substitute for good pages. If the content it points to is thin, the file simply routes a model to thin content faster.

Who actually reads it in 2026

This is the section worth reading twice, because the honest answer is more interesting than either the hype or the dismissal.

Google Lists llms.txt among the tactics site owners can ignore. John Mueller has compared it to the keywords meta tag. Gary Illyes and Amir Taboul confirmed at Search Central Live that Google is not pursuing it. It plays no part in AI Overviews or AI Mode. Not used

Anthropic Recommends llms.txt in its guidance on writing for agents, on the basis that coding agents work better with a curated documentation map than with a site they have to reconstruct. Recommends

OpenAI Publishes llms.txt files for its own developer properties, including the Agents SDK and the Agentic Commerce Protocol. Publishing is not the same as consuming, but it signals the format is taken seriously. Publishes

Perplexity Has been observed surfacing llms.txt content independently of its normal retrieval pipeline. Suggestive rather than confirmed, but it is the clearest signal of consumer-side use so far. Observed

Positions as publicly stated to June 2026. Vendor guidance changes; check before you rely on it.

Notice the shape of that table. The company that dominates search says no. The companies building agents say yes, or at least behave as though the format matters to them.

That is the real distinction, and it is the one to build your decision on. llms.txt has found a genuine use in agent and developer contexts, and has not found one in consumer AI search.

What the evidence shows

Wix Studio’s AI Search Lab found in October 2025 that Google had indexed somewhere between 30,000 and 60,000 llms.txt files globally. Crawling is clearly happening, whatever Google Search says about using the contents.

What does not exist, at the time of writing, is a credible study showing that publishing llms.txt increases how often a site is cited by ChatGPT, Perplexity, Claude or anyone else. Not one. The case studies circulating tend to be single sites that changed several things at once and attributed the result to the cheapest change.

That absence of evidence is not proof the file does nothing. It is a reason to hold the claim loosely and to keep your expectations proportionate to a file that takes an hour to write.

How to write one properly

The format is deliberately simple. It is Markdown, and it has a defined structure.

# Indexify

> Practical AI training, AI consultancy and managed digital growth
> for UK organisations, backed by technical SEO, analytics and
> conversion expertise.

Indexify works with UK businesses that need AI to produce a
measurable commercial result rather than a pilot that stalls.
Everything below is written and maintained by hand.

## Services

- [AI Readiness Audit](https://indexify.co.uk/services/ai-readiness-audit):
  A structured assessment of whether an organisation can adopt AI safely,
  covering data, skills, governance and process. Fixed price.
- [AI Search Visibility](https://indexify.co.uk/services/ai-search-visibility):
  Making a brand findable and citable inside ChatGPT, Perplexity and
  Google AI Overviews.
- [Technical SEO](https://indexify.co.uk/services/technical-seo):
  Crawl, index and performance work for sites where the fundamentals
  are holding rankings back.

## Guides

- [What an AI Consultant Costs in the UK](https://indexify.co.uk/guides/ai-consultant-cost-uk):
  Benchmarked day rates and project prices for UK AI consultancy,
  with our own pricing included.
- [Technical SEO Audit Checklist](https://indexify.co.uk/guides/technical-seo-audit-checklist):
  The checklist we run on client sites, in the order we run it.

## Optional

- [About](https://indexify.co.uk/about): Who we are and how we work.
- [Contact](https://indexify.co.uk/contact): How to reach us.

The rules that matter:

One H1, and it is your site or organisation name. Nothing else.

A blockquote immediately after it. This is the summary a model is most likely to lift. Write it as though it is the only line anyone reads, because it might be.

H2 sections grouping links. Name them for what they contain, not for your internal taxonomy.

Every link gets a description. A bare URL wastes the entire point of the file. The description is the routing instruction.

An ## Optional section for anything that can be skipped. This section carries specific meaning in the spec: it marks content a model may drop when it is short of context. Use it, and put your genuinely secondary pages there.

llms.txt and llms-full.txt

You will see llms-full.txt referenced alongside it. The distinction is simple:

• llms.txt is the index. Links and descriptions, nothing more.
• llms-full.txt is the whole thing. Your entire documentation set concatenated into one Markdown file, so a model can ingest it in a single request.

llms-full.txt makes sense for documentation sites, developer platforms and API references, where an agent genuinely benefits from having everything at once. For most marketing sites it is an enormous file that nobody requests. Start with llms.txt. Add the full version only if you are running a documentation property.

Five mistakes worth avoiding

Dumping your sitemap into it. Two hundred URLs with no descriptions is not a curated map, it is a sitemap in a different file extension. Twenty well-described links beat two hundred bare ones.

Writing descriptions for search engines. Keyword-stuffed descriptions read badly to a model that is trying to decide what to fetch. Write them the way you would brief a new colleague.

Letting it go stale. A file that points to three pages you deleted in January is actively worse than no file. Put a review date in your calendar.

Publishing it as HTML. It must be served as plain text or Markdown at /llms.txt. If your framework wraps it in a page template, the file is broken. Test it in a browser and check the raw response.

Expecting it to fix visibility. If you are not being cited by AI systems, the cause is almost never a missing text file. It is usually that your content does not answer the question cleanly, is not corroborated anywhere else, or is not being crawled at all.

So should you publish one?

Here is the decision, honestly framed.

1. Clear yesDocumentation sites, developer platforms, API references, and any product a coding agent will be asked to work with. This is the use case with real vendor support behind it.
2. Reasonable yesSites with a large content library where an hour of curation genuinely clarifies what matters. The file also makes a useful internal artefact: it forces you to name your twenty best pages.
3. Do it, expect nothingOrdinary marketing and brochure sites. It costs an hour and cannot hurt. Just do not put it on a roadmap as an AI visibility initiative, and do not report it as one.
4. Clear noAnyone about to spend a fortnight on it, or to skip fixing thin content, slow pages or missing schema in order to build one. The opportunity cost is the only real risk this file carries.

How to tell whether it is doing anything

If you publish one, measure it properly rather than assuming.

Check your server logs, not your analytics. AI crawlers do not execute JavaScript, so they are invisible to most analytics tools. Filter your access logs for requests to /llms.txt and note which user agents appear. If nothing has requested it in ninety days, you have your answer for your site.

Track citations separately. Run a fixed set of prompts against ChatGPT, Perplexity and Google AI Mode every month, record whether you appear, and keep the record. That baseline is worth far more than the file itself, because it tells you whether anything you do is working.

Do not change two things at once. If you publish llms.txt in the same week you rewrite your service pages, you will never know which one moved the needle.

The short version

llms.txt is a well-designed solution to a real problem, adopted enthusiastically by the agent ecosystem and declined by the largest search engine. It takes about an hour, cannot damage anything, and has no proven effect on how often consumer AI tools cite you.

Write one if you run documentation, or if the curation exercise is useful to you in its own right. Then go and spend the rest of the week on the things that demonstrably do affect whether AI systems cite you: content that answers a question completely, corroboration from sources other than your own site, clean crawlability, and structured data that says plainly what you are.

If you would like an independent view of how visible your business currently is inside AI answers, our AI search visibility service starts with a measured baseline rather than a checklist.