llms.txt Explained: What It Is and How to Add It

What Is llms.txt?

llms.txt is a plain-text file, written in Markdown, that a website publishes at its root address — yoursite.com/llms.txt — to give AI systems a curated index of its most important pages. Each entry typically pairs a page title, a one-line description, and a link, organised under headings. The idea draws a direct analogy to robots.txt: a lightweight file that any crawler can fetch instantly, without needing to parse a full HTML page.

The format has two variants in circulation. Standard llms.txt is an index — titles, links, and short descriptions. A longer companion, llms-full.txt, embeds the entire content of the linked pages in one file, so an AI agent can ingest everything without a second fetch. Companies serving Markdown instead of HTML have reported up to 10x token reductions, which translates directly into faster, cheaper, more accurate agent behaviour — the core technical argument behind the format.

It is worth being precise about what the file is not. It cannot block or restrict any crawler, unlike robots.txt. It does not opt content out of AI training. It is not a ranking factor, an indexing signal, or a form of structured data recognised by Google. It is, at most, a suggestion — one that some tools may read and most currently do not.

Who Invented llms.txt and Why?

Jeremy Howard, co-founder of Answer.AI and fast.ai, published the llms.txt proposal on 3 September 2024 at answer.ai and the companion site llmstxt.org. The problem he set out to solve had nothing to do with search rankings or citation volume. Context windows of LLMs were too small for complete websites, and converting HTML — with its navigation, advertising, and JavaScript — into clean LLM-friendly text was laborious and error-prone, particularly for developers whose AI coding assistants needed fast, accurate access to API documentation.

Howard’s own FastHTML project served as the reference implementation — a Python framework with technical documentation, precisely the use case the format was designed for. The proposal spread quickly through the developer-tools ecosystem, not general marketing or SEO circles, because that is where the underlying pain point — bloated HTML slowing down AI coding agents — was most acute.

That origin story matters. It explains why the file’s strongest, most demonstrable value today remains in developer documentation rather than commercial marketing content, and why its adoption trajectory has followed the tooling ecosystem more than the SEO industry.

Does llms.txt Actually Improve AI Search Visibility?

The evidence available in mid-2026 says no, at least not in any measurable way for citation frequency in ChatGPT, Perplexity, Gemini, or Google’s AI Overviews. Three independent data points converge on this conclusion.

First, Google has said so directly. At the Search Central Deep Dive event in Bangkok in July 2025, Gary Illyes stated plainly that Google does not support llms.txt and has no plans to. John Mueller went further on Reddit, writing that none of the AI services have said they’re using llms.txt, and that server logs show they don’t even check for it — comparing it directly to the keywords meta tag, a signal Google stopped using decades ago because it was self-declared and easily gamed. Google’s own AI optimisation guidance, updated 15 June 2026, states directly that publishers don’t need to create machine-readable files, AI text files, markup, or Markdown to appear in Google Search, including its generative AI features, because Google Search itself doesn’t use them.

Second, server-log analysis backs this up. Ahrefs analysed roughly 137,000 websites with an llms.txt file and found that 97% were never fetched by any bot at all. Separately, an analysis of over 515 million AI bot traffic events found the share of requests touching /llms.txt from the crawlers that actually drive citations — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended — statistically negligible.

Third, the correlation studies come back flat. SE Ranking analysed roughly 300,000 domains, ran both a statistical model and an XGBoost classifier to test whether the file correlated with AI citations, and found no measurable effect. There is a genuine exception worth noting: Perplexity is the one major provider that has said it retrieves the file to help prioritise its reading, though even there the practical citation lift has not been independently confirmed at scale.

How Is llms.txt Different from robots.txt?

The two files solve opposite problems, and confusing them is the most common mistake site owners make. Robots.txt governs access — it tells compliant crawlers which parts of a site they may or may not fetch, and it is a genuine, decades-old web standard, formalised as RFC 9309. llms.txt, by contrast, is a proposal for navigation and summarisation — it doesn’t block anything and can’t; it simply points to content the site owner considers important.

Note: robots.txt compliance and adoption figures reflect established web-crawling standards; llms.txt figures reflect 2026 industry research cited in the Sources section below, as no formal registry tracks the file.

The practical implication: robots.txt still deserves careful attention, especially given that a meaningful share of sites accidentally block major AI crawlers at the CDN layer while their robots.txt says allow, meaning the two layers must agree. llms.txt does not carry that same operational weight.

Should You Still Add llms.txt to Your Website?

Yes, as low-cost hygiene, but not as a strategic AI-visibility investment. The honest case for adding the file rests on three points: it costs very little to produce, it carries no downside, and it has documented value for one specific audience — coding agents and documentation-heavy sites.

Anthropic’s own engineering documentation references flat llms.txt files in the context of developer documentation workflows, and Mintlify’s rollout of llms.txt support across all docs sites it hosts brought the file to thousands of sites overnight, including Anthropic and Cursor; Fern, GitBook, Vercel Docs, Supabase Docs, Yoast, and Rank Math now ship it by default. If a business runs a SaaS product, an API, or technical documentation that developers query through Cursor, Claude Code, or similar tools, llms.txt has a plausible, demonstrable use case.

For a hotel, restaurant, wellness brand, or service business, the calculus is different. The file will do nothing measurable for AI Overviews or ChatGPT citation volume on current evidence. What actually drives inclusion in AI answers is structured, well-organised content the crawlers already read directly: clear headings that match real search questions, sourced facts and figures, consistent entity naming, and technical performance that lets crawlers render the page cleanly. Kafkasque builds that foundation through web design & development and on going SEO work, because those are the levers with actual evidence behind them.

How Do You Add llms.txt to a WordPress Site?

The fastest method is a plugin; the more durable method is a static file uploaded via FTP or hosting file manager. Both take under an hour.

Plugin method. Several SEO plugins — including recent versions of Yoast and Rank Math — now generate an llms.txt automatically from existing page titles, meta descriptions, and site structure. Install or update the plugin, enable the llms.txt module in its settings, and the file becomes live at yoursite.com/llms.txt without any code.

Manual method. For more control over what gets included:

1. Create a plain-text file named exactly llms.txt.
2. Structure it in Markdown: an H1 with the site or brand name, a one-line summary blockquote, then H2 sections grouping links by category (for example, “Services,” “Case Studies,” “Contact”).
3. Under each link, add a short, factual one-line description — not marketing copy.
4. Upload the file to the site’s root directory via FTP, cPanel File Manager, or a hosting dashboard’s file manager, so it resolves at yoursite.com/llms.txt with no subfolder.
5. Verify it loads correctly by visiting the URL directly in a browser and confirming it renders as plain text, not a 404 or an HTML wrapper.
6. Re-check the file every time major pages are added, renamed, or retired, since a stale index is worse than no index at all.

For technical documentation sites, an llms-full.txt can be generated the same way, with full page content embedded rather than links — useful if a coding agent or internal RAG pipeline is the intended reader. For a WordPress build handled end-to-end, Kafkasque’s web maintenance and optimisation service covers this kind of file alongside broader technical upkeep.

What Mistakes Should You Avoid with llms.txt?

The most common and costly mistake is treating the file as a sitemap. Listing every URL on the site with no meaningful descriptions defeats the purpose of a curated index and signals low effort to the small number of tools that do read it. A second mistake is letting the file go stale — pointing to pages that have moved, been merged, or no longer exist. A third is paying an agency for llms.txt as a recurring service line on the promise of an AI-citation lift; the current data does not support that framing, and any vendor selling it as such is overselling a low-cost technical task.

What Does the Data Say About Real-World Adoption in 2026?

Adoption has grown steadily but stayed far short of universal, and the pattern of who adopts it tells its own story. A SE Ranking study of 300,000 domains found a 10.13% adoption rate — roughly one in ten sites, eighteen months after the proposal was first published. Among Fortune 500 companies specifically, 7.4% — 37 of 500 — had shipped an llms.txt by 31 March 2026, with developer-documentation sites adopting far faster than the general commercial web, because coding agents remain the one consumer that demonstrably reads these files today.

This uneven pattern is the clearest read on where the format currently sits: genuinely useful infrastructure for a narrow, technical audience; largely inert for everyone chasing a citation boost in consumer-facing AI answers.