llms.txt Advisor

What this is

llms.txt is a plain-text Markdown file a site publishes at the root of its own domain (/llms.txt) so AI engines have a short, curated map of what the site is and which pages matter, in the shape the llmstxt.org spec defines: a title, an optional summary, and sections of links. llms.txt Advisor fetches your project's live /llms.txt, checks it against that spec with a fixed, deterministic rule set, then has a model draft a compliant replacement for you to publish yourself (api/services/llms_txt/validate.py#validate_llms_txt, api/services/llms_txt/service.py#run_llms_txt_audit). It is gated on your plan; see Pricing.

The checklist: eight rules against the spec, and one note

For a file that has content, the checklist runs in the order below. Eight of the nine entries run every time; the ## Optional note is the exception, appearing only for a file that has such a section, so a file without one shows eight. Only the first two can fail; everything else tops out at a warning (api/services/llms_txt/validate.py#validate_llms_txt):

• File has content: fails if nothing was fetched or the file is empty.
• Has an H1 title: fails unless the very first line is a single # heading.
• Exactly one H1: warns if more than one H1 heading is present.
• Has a blockquote summary: warns unless one of the five lines right after the title starts with >.
• Sections use H2: warns if any heading goes deeper than H2.
• Link items well-formed: warns if a bullet under a section is not a markdown link such as [label](https://example.com/page).
• Link URLs valid: warns if a link's URL is empty or uses an unsafe scheme such as javascript:.
• Has an Optional section: informational only, and the one entry that is not always there: it appears only when a heading titled ## Optional is present, and it is never scored.
• No raw HTML: warns if a raw HTML tag appears outside a blockquote line.

A file that is empty or was not found is a special case: only the first two checks run and both fail, rather than the rest being shown as passing something they never had a chance to look at.

What "compliant" means, and what it does not check

Because only "File has content" and "Has an H1 title" can fail, compliant here means exactly that: the file exists and opens with a # title line. Every other rule can raise a warning without changing that verdict. The checklist is a structural check against the spec's shape, not the whole spec: it does not follow any linked URL to confirm the page still exists, and it does not judge whether a summary or a link description is any good. A sibling file some sites publish, llms-full.txt, is checked only for whether a file is served at that address, shown on screen alongside the file's response status; that presence check is not part of the checklist and does not affect whether the file is compliant (api/services/llms_txt/fetch.py#llms_full_txt_exists).

Running one, on demand or weekly

"Run a check" starts an audit immediately if the project owner's plan includes the advisor. Editors and owners can start one; a viewer sees the button disabled, both the one in the page header and the second one this screen's empty state shows when a project has no audits yet, each with a role notice above it saying why (api/routers/llms_txt_advisor.py#trigger_audit, #_trigger_block_reason, #TRIGGER_COOLDOWN_HOURS, frontend/src/routes/(app)/llms-txt-advisor/+page.svelte#writable). Separately, the project's owner can turn on a weekly automatic check from the same screen; turning it on is gated on the same plan check as running one on demand (api/routers/projects.py#update_project). An editor sees that "Check weekly" checkbox too, rather than it being left off the screen, but it renders disabled with its own notice that the schedule is owner-only (frontend/src/routes/(app)/llms-txt-advisor/+page.svelte#owner). Once it is on, a systemd timer fires every Tuesday at 05:00 UTC and enqueues one audit for every opted-in project that is itself active, whose owner is active and on a paid plan, and that has no audit already running or started within the last 7 days, so running one yourself pushes that project's next automatic check back rather than doubling up (discoveredby-cli@enqueue-llms-txt-advisor-refresh.timer, api/tasks.py#_enqueue_llms_txt_advisor_refresh_for_eligible_projects, #LLMS_TXT_ADVISOR_REFRESH_INTERVAL_DAYS, #generate_llms_txt_audit_task).

The file it writes, and what you do with it

The one thing this feature does with your site is read from it: fetching your current /llms.txt is a GET request, nothing more (api/services/llms_txt/fetch.py#fetch_llms_txt). The compliant file a model drafts afterward, proposed_llms_txt, is saved to the audit row and rendered on screen in a text box with a Copy button (api/services/llms_txt/service.py#run_llms_txt_audit, frontend/src/routes/(app)/llms-txt-advisor/+page.svelte#copied).

Nothing is written to your site by this feature, or by anything else in the product. Putting the proposed text at the root of your own domain is a step you carry out yourself, outside DiscoveredBy, once you have copied it from the screen.

The fetch is SSRF-guarded

Because the address being fetched comes from your project's own URL, every hop is checked before DiscoveredBy connects. The fetch refuses anything that is not a plain http/https URL, refuses a literal IP address unless it is globally routable (so no private, loopback, link-local, or reserved range), and refuses localhost and other single-label hostnames (api/services/urls.py#is_public_http_url). It then resolves the hostname's DNS and refuses to connect if that resolves to a private, loopback, link-local, multicast, reserved, or unspecified address, which catches a public-looking hostname that actually points inward (#resolved_url_is_public, api/services/llms_txt/fetch.py#_is_public). Both checks run again on every redirect hop, not just the first request, and the fetch gives up after 3 redirects and caps what it reads at 512,000 bytes (#fetch_llms_txt, #_MAX_REDIRECTS, #LLMS_TXT_MAX_BYTES). This guards this one fetch; it is not a general claim about the product's security posture.

Rule-derived, and model-written

The checks above are plain code with no model involved: the same input always produces the same checklist. What is written afterward, the summary, the suggested fixes, and the proposed replacement file, is a model's prose and draft text, built from your project's identity and the checklist results and framed to it as data, not instructions (api/services/llms_txt/service.py#build_review_prompt, #REVIEW_SCHEMA). The fetch and the checklist are saved before that model call runs, so if the model phase fails, the fetch result and the checklist still show on screen; only the written summary, suggestions, and proposed file are missing, and running the check again retries the whole thing (#run_llms_txt_audit).

What's on the screen

The headline reads Compliant, Not compliant, or No llms.txt, depending on whether the file was reachable and what the checklist found (frontend/src/routes/(app)/llms-txt-advisor/+page.svelte#compliant). Below it, every check that ran is listed with its level; the warning and failure counts shown are drawn from that same list, not written separately (#checks, #failCount, #warnCount). This documentation site is generated the same way discoveredby.ai's own /llms.txt is, both built from a single index rather than hand maintained, though this site's own file is not run through the advisor.

Related

• Weekly report: the other periodic artifact on the account, with no model involved anywhere in it
• Growth Advisor: the other advisor whose output is model-written rather than derived
• What is GEO?: the broader idea of making a site legible to AI systems that this file is one piece of