llms.txt is a proposed standard: a Markdown file placed at the root of your site, at /llms.txt, that lists your most important pages in a form a language model can read cheaply. It was proposed in 2024 and is meant to help AI assistants understand a site without crawling and cleaning all of its HTML. It is a proposal, not an official standard.

Do AI models actually use llms.txt?

There is no confirmed use of it yet. Google has said publicly that Search does not use llms.txt, and no major AI provider such as OpenAI, Anthropic or Perplexity has committed to reading it. Sites are adopting it on the publishing side, but that is not the same as any assistant committing to consult it, so treat publishing one as optimistic rather than proven.

Where do I put the llms.txt file?

Place it at the root of your domain so it loads at yoursite.com/llms.txt, the same location convention as robots.txt. It is a plain text file written in Markdown, so any web server that can serve a static text file can serve it. There is an optional companion file, llms-full.txt, that inlines the full content of the listed pages instead of linking to them.

What is the difference between llms.txt and robots.txt?

robots.txt controls which paths a crawler is allowed to fetch — it is about permission. llms.txt hand-picks the few pages you most want an AI assistant to read and understand — it is about emphasis. They are different jobs and a site can have both: robots.txt is honored by search and AI crawlers today, while llms.txt is a newer proposal that nobody has confirmed reading.

Is llms.txt worth creating?

It is worth creating as cheap insurance, not as a priority. It takes about ten minutes, cannot hurt you, and puts you ahead if the convention is adopted. It is not worth prioritising over the things that actually decide AI visibility: allowing AI crawlers in robots.txt, serving your content in the HTML rather than via JavaScript, and adding structured data.

How do I write a good llms.txt file?

Write it by hand and keep it short. Start with a level-one heading naming your site and an optional one-line summary, then group your best pages under level-two headings with a bulleted list of links, each with a short note. List only the pages you would genuinely want an assistant to understand your site from — a file auto-generated from every URL is just a worse sitemap.

What is llms.txt?

What the file is, the format it uses, how to publish one — and the honest position on whether Google or any AI provider actually reads it yet.

By Max7 min read

llms.txt is a proposed standard: a Markdown file you place at the root of your site, at /llms.txt, that lists your most important pages in a form a language model can read cheaply. It is meant to do for AI assistants what a sitemap does for search crawlers — point them at the pages that matter. The honest part, which most guides skip, is that no major AI company has committed to reading it, and Google has said Search does not use it at all.

What the file actually is

The idea was proposed in September 2024 by Jeremy Howard of Answer.AI and published at llmstxt.org. The problem it tries to solve is real: an assistant that wants to understand your site has to fetch your HTML, strip out the navigation, scripts and markup, and hope the content survives. A short Markdown file listing your key URLs, with a sentence about each, is far cheaper for a model to consume than crawling and cleaning a hundred pages.

So llms.txt is a curated table of contents written for machines. It is not generated from your pages the way a sitemap often is; it is a short, hand-picked list of the things you would most want an assistant to know about, in the order you would want it to meet them.

What goes in it

The format is plain Markdown with a loose structure. It opens with a level-one heading naming your site, then an optional blockquote summarising what the site is. After that come level-two headings that group your links — “Docs”, “Guides”, “Products” — each followed by a bulleted list of links, and each link may carry a short note after a colon explaining what it is.

A section headed Optional is treated as exactly that: material an assistant can skip when it is working to a tight context budget. There is also a companion convention, llms-full.txt, which inlines the full text of those pages into one file rather than linking out to them, for models that would rather read once than fetch repeatedly.

Does anything actually read it?

This is the question that decides whether the file is worth your afternoon, and the answer is: not in any confirmed way, yet. It is a proposal, not a specification anyone is obliged to follow. Google has said publicly that Search does not use llms.txt, and no major AI provider — OpenAI, Anthropic, Perplexity — has committed to reading it either.

The contrast worth drawing is with robots.txt and the XML sitemap. Those are honored because the companies that crawl the web agreed to honor them, over years, in public. llms.txt has adoption from the tools that generate it and a growing list of sites that publish it, but adoption on the writing side is not the same as adoption on the reading side. Until an assistant says it consults the file, publishing one is an act of optimism.

Should you publish one?

Yes, with the right expectations. It costs about ten minutes, it cannot hurt you, and if the convention is adopted you are already there. Treat it as cheap insurance. The mistake is treating it as the thing standing between you and being quoted, and spending effort on it that belongs elsewhere.

“Elsewhere” means the conditions that actually decide whether an AI can read your page: whether you block AI crawlers, whether your content is in the HTML rather than injected by JavaScript, and whether your facts are in structured data. The AI visibility checker guide covers what those are, and the generative engine optimization checklist puts them in priority order. llms.txt sits near the bottom of that list for a reason.

How to create one

Write it by hand — the whole value is in the curation, and a file auto-generated from every URL you have is just a sitemap with worse formatting. List the pages you would genuinely want an assistant to understand your site from: your main explainers, your best reference pages, the things that describe what you do. Give each a one-line note. Save it as plain text at /llms.txt and keep it in sync when those pages change.

Our own file, at /llms.txt, is built from the same registry that drives the site’s navigation, so it cannot list a page that no longer exists. That is a nice property to aim for, but it is not required — a static file you update by hand is perfectly valid, and for most sites it is the right amount of effort.

llms.txt, robots.txt and sitemaps are three different jobs

They get confused because all three are files at your site root that talk to automated visitors, but they do different things. robots.txt controls which paths a crawler may fetch — permission. An XML sitemap lists every URL you want discovered — coverage. llms.txt hand-picks the few pages you most want an assistant to read and understand — emphasis. A site can and often should have all three, and none of them replaces another.

How to check yours

Load yoursite.com/llms.txt in a browser. It should return plain Markdown with a heading and real, working links — not a 404, and not your HTML homepage served by a catch-all route. If you want the bigger picture of whether AI assistants can reach your site at all, the free website check reports your AI crawler access, rendering and structured data together, because those matter far more than this file does.

Frequently asked questions

What is llms.txt?: llms.txt is a proposed standard: a Markdown file placed at the root of your site, at /llms.txt, that lists your most important pages in a form a language model can read cheaply. It was proposed in 2024 and is meant to help AI assistants understand a site without crawling and cleaning all of its HTML. It is a proposal, not an official standard.
Do AI models actually use llms.txt?: There is no confirmed use of it yet. Google has said publicly that Search does not use llms.txt, and no major AI provider such as OpenAI, Anthropic or Perplexity has committed to reading it. Sites are adopting it on the publishing side, but that is not the same as any assistant committing to consult it, so treat publishing one as optimistic rather than proven.
Where do I put the llms.txt file?: Place it at the root of your domain so it loads at yoursite.com/llms.txt, the same location convention as robots.txt. It is a plain text file written in Markdown, so any web server that can serve a static text file can serve it. There is an optional companion file, llms-full.txt, that inlines the full content of the listed pages instead of linking to them.
What is the difference between llms.txt and robots.txt?: robots.txt controls which paths a crawler is allowed to fetch — it is about permission. llms.txt hand-picks the few pages you most want an AI assistant to read and understand — it is about emphasis. They are different jobs and a site can have both: robots.txt is honored by search and AI crawlers today, while llms.txt is a newer proposal that nobody has confirmed reading.
Is llms.txt worth creating?: It is worth creating as cheap insurance, not as a priority. It takes about ten minutes, cannot hurt you, and puts you ahead if the convention is adopted. It is not worth prioritising over the things that actually decide AI visibility: allowing AI crawlers in robots.txt, serving your content in the HTML rather than via JavaScript, and adding structured data.
How do I write a good llms.txt file?: Write it by hand and keep it short. Start with a level-one heading naming your site and an optional one-line summary, then group your best pages under level-two headings with a bulleted list of links, each with a short note. List only the pages you would genuinely want an assistant to understand your site from — a file auto-generated from every URL is just a worse sitemap.