llms.txt is a plain-text file you publish at the root of your site (at /llms.txt) that tells AI crawlers what your site is about and points them to the pages that matter most, in a format language models can read cleanly.
When an AI engine wants to understand your site, it usually has to crawl raw HTML: navigation, scripts, cookie banners, and boilerplate wrapped around the content that actually matters. llms.txt cuts through that. Defined at llmstxt.org, it is a short Markdown file that hands engines a curated map of your site: a title, a one-line summary, and lists of your most important pages grouped under headings. It sits alongside robots.txt and sitemap.xml as part of your site's machine-readable layer, but where those speak to search crawlers, llms.txt speaks to language models.
Why llms.txt matters now
- AI engines increasingly retrieve live content, and a clean map makes it easier for them to find and cite the right page.
- It lets you decide which pages represent you, instead of leaving an engine to guess from your homepage markup.
- Writing one forces clarity: you have to name your most citation-worthy pages and describe each in a sentence.
- It is low-cost to add and positions you early, as adoption across engines grows.
What goes in an llms.txt file
The spec is deliberately simple, but it has a strict shape. A well-formed file has a single H1 title, an optional summary in a blockquote, and one or more H2 sections, each holding a list of links with short descriptions:
# Acme Analytics
> Acme Analytics is privacy-first product analytics for SaaS teams.
## Docs
- [Quickstart](/docs/quickstart): get set up in five minutes
- [API reference](/docs/api): the full REST and webhook reference
## Product
- [Pricing](/pricing): plans, limits, and what each tier includes
- [Security](/security): how we handle and protect your data
## Optional
- [Changelog](/changelog): release notes
- One H1 title: the name of your site or product. This is the only strictly required element.
- A summary blockquote: a single line, starting with
>, describing what you do. - H2 sections of links: group related pages under headings, each link followed by a short description.
- An Optional section: a special heading for pages an engine can safely skip when it needs shorter context.
Get that shape wrong, miss the H1, drop the summary, malform the links, and an engine falls back to guessing. Getting it right is mechanical, which is exactly why it is worth checking against the spec rather than eyeballing it.
How to create and maintain one
- List your key pages. Pick the handful of pages you most want cited: docs, pricing, product, comparisons, and core guides.
- Write it in the spec's format. One H1, a summary blockquote, H2 sections of links with a short description on each.
- Publish it at /llms.txt. Serve it as plain text from the root of your domain.
- Keep it current. Re-check it whenever your site changes, so the map never points at pages that moved or went stale.
This is what the DiscoveredBy llms.txt Advisor automates: it fetches your live file, grades it against every rule in the llmstxt.org spec, explains what to fix in plain language, and generates a compliant llms.txt built around your key pages. Related reading: What is GEO?, What is AEO?, and On-Page Optimization for AI.
Frequently asked questions
Where does the llms.txt file go?
At the root of your domain, served at /llms.txt as plain-text Markdown, the same place robots.txt and sitemap.xml live. Some sites also publish an /llms-full.txt with the expanded content of those pages inlined, for engines that want the full context in one fetch.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which paths they are allowed to access. llms.txt does the opposite job: it curates what matters and points language models straight to your most useful pages, with a short description of each. They complement each other rather than compete.
Do AI engines actually read llms.txt?
It is an emerging standard, and adoption is still growing, so support varies by engine and is not guaranteed today. The upside is that it costs very little to add, it positions you as more engines adopt it, and the exercise of writing one forces useful clarity about which pages you most want cited.
How do I know if my llms.txt is valid?
Check it against the llmstxt.org spec: a single H1 title, a summary blockquote, and H2 sections containing well-formed links. DiscoveredBy's llms.txt Advisor audits your live file against every rule in the spec, tells you what to fix, and generates a compliant file built around your key pages.