All documentation

Advisors

llms.txt Advisor

A weekly audit of your live llms.txt against the spec, with a compliant file written out for you to publish.

llms.txt Advisor screen showing a compliant result with every spec check passing

What this is

llms.txt is a plain-text Markdown file a site publishes at the root of its own domain (/llms.txt) so AI engines have a short, curated map of what the site is and which pages matter, in the shape the llmstxt.org spec defines: a title, an optional summary, and sections of links. llms.txt Advisor fetches your project's live /llms.txt, checks it against that spec with a fixed, deterministic rule set, then has a model draft a compliant replacement for you to publish yourself (api/services/llms_txt/validate.py#validate_llms_txt, api/services/llms_txt/service.py#run_llms_txt_audit). It is gated on your plan; see Pricing.

The checklist: eight rules against the spec, and one note

For a file that has content, the checklist runs in the order below. Eight of the nine entries run every time; the ## Optional note is the exception, appearing only for a file that has such a section, so a file without one shows eight. Only the first two can fail; everything else tops out at a warning (api/services/llms_txt/validate.py#validate_llms_txt):

  • File has content: fails if nothing was fetched or the file is empty.
  • Has an H1 title: fails unless the very first line is a single # heading.
  • Exactly one H1: warns if more than one H1 heading is present.
  • Has a blockquote summary: warns unless one of the five lines right after the title starts with >.
  • Sections use H2: warns if any heading goes deeper than H2.
  • Link items well-formed: warns if a bullet under a section is not a markdown link such as [label](https://example.com/page).
  • Link URLs valid: warns if a link's URL is empty or uses an unsafe scheme such as javascript:.
  • Has an Optional section: informational only, and the one entry that is not always there: it appears only when a heading titled ## Optional is present, and it is never scored.
  • No raw HTML: warns if a raw HTML tag appears outside a blockquote line.

A file that is empty, or an address where the site answered without serving one (a 404, say), is a special case: only the first two checks run and both fail, rather than the rest being shown as passing something they never had a chance to look at.

A fetch that gets no HTTP answer at all is different again, because it says nothing about whether the file exists. When the request times out, the connection fails, the address or a redirect from it points at a host that is not on the public internet, or a redirect cannot be followed, the check is marked failed with the reason, no checklist runs and no replacement is drafted (api/services/llms_txt/service.py#run_llms_txt_audit, api/services/llms_txt/fetch_problem.py#classify_fetch_error). The screen reads Not reachable with the reason ("Timed out", "Connection failed", "Address refused" or "Broken redirect"), not "No llms.txt". A check from before this distinction that recorded no HTTP answer reads "No answer".

What "compliant" means, and what it does not check

Because only "File has content" and "Has an H1 title" can fail, compliant here means exactly that: the file exists and opens with a # title line. Every other rule can raise a warning without changing that verdict. The checklist is a structural check against the spec's shape, not the whole spec: it does not follow any linked URL to confirm the page still exists, and it does not judge whether a summary or a link description is any good. A sibling file some sites publish, llms-full.txt, is checked only for whether a file is served at that address, shown on screen alongside the file's response status; that presence check is not part of the checklist and does not affect whether the file is compliant (api/services/llms_txt/fetch.py#llms_full_txt_exists).

Running one, on demand or weekly

"Run a check" starts an audit immediately if the company's plan includes the advisor. On a plan without it, the screen says so before you click, and checks run before stay readable. One check a day is the limit, counted from when the last one started, but only a check that succeeded or is still running uses it up: a failed check does not (api/services/advisor_limits.py#takes_the_day, #COOLDOWN_HOURS). Every start you make, or the weekly refresh makes, counts toward a second limit, whatever became of it: at most three checks can start in any 24 hours, so a check that keeps failing cannot keep spending model time (#cap_ends, #MAX_RUNS_PER_DAY). Weekly automatic checks count toward both limits like the ones you run; a check a DiscoveredBy admin starts while helping you counts toward neither (#counts_toward_limits). Once either limit is reached, the screen names the time the next can start ("The next check can start after ...") instead of letting the click fail, and a failed or unfinished check says when a new one can start, promising "now" only when both the limits and your plan are known to allow it (#next_run_at, frontend/src/routes/(app)/llms-txt-advisor/+page.svelte#retryLine). While a check runs, the page re-reads its checks every 10 seconds while the tab is visible, so the result appears without a reload. A check still marked running past the worker's time limit has stopped: it reads "This check did not finish", and like a failed one it neither blocks a new check nor uses up the day, though it is still one of the three starts (api/services/stale_runs.py#is_abandoned). Editors and company members can start one; a viewer sees the button disabled, both the one in the controls above the results and the second one this screen's empty state shows when a project has no audits yet, each with a role notice above it saying why (api/routers/llms_txt_advisor.py#trigger_audit, api/services/advisor_limits.py#trigger_block_reason, frontend/src/routes/(app)/llms-txt-advisor/+page.svelte#writable). Separately, a member of the project's company can turn on a weekly automatic check from the same screen; turning it on is gated on the same plan check as running one on demand (api/routers/projects.py#update_project). An editor sees that "Check weekly" checkbox too, rather than it being left off the screen, but it renders disabled with its own notice that only company members can change the schedule (frontend/src/routes/(app)/llms-txt-advisor/+page.svelte#owner). If the setting cannot be read, the checkbox stays disabled with a note rather than reading as off, and on a plan without the advisor it can only be switched off. Once it is on, a systemd timer fires every Tuesday at 05:00 UTC and enqueues one audit for every opted-in project that is itself active, whose company is active and on a paid plan, and that has no audit already running or started within the last 6 days (a day short of a week, so last Tuesday's check never holds up this Tuesday's), so running one yourself pushes that project's next automatic check back rather than doubling up. A failed check counts here too, on demand or weekly, so a failed weekly check is not retried until a later Tuesday; running one yourself is the way to try again sooner (discoveredby-cli@enqueue-llms-txt-advisor-refresh.timer, api/tasks.py#_enqueue_llms_txt_advisor_refresh_for_eligible_projects, #WEEKLY_REFRESH_MIN_GAP, #generate_llms_txt_audit_task).

The file it writes, and what you do with it

The one thing this feature does with your site is read from it: fetching your current /llms.txt is a GET request, nothing more (api/services/llms_txt/fetch.py#fetch_llms_txt). The compliant file a model drafts afterward, proposed_llms_txt, is saved to the audit row and rendered on screen in a text box with a Copy button (api/services/llms_txt/service.py#run_llms_txt_audit, frontend/src/routes/(app)/llms-txt-advisor/+page.svelte#copied).

Nothing is written to your site by this feature, or by anything else in the product. Putting the proposed text at the root of your own domain is a step you carry out yourself, outside DiscoveredBy, once you have copied it from the screen.

The fetch is SSRF-guarded

Because the address being fetched comes from your project's own URL, every hop is checked before DiscoveredBy connects. The fetch refuses anything that is not a plain http/https URL, refuses a literal IP address unless it is globally routable (so no private, loopback, link-local, or reserved range), and refuses localhost and other single-label hostnames (api/services/urls.py#is_public_http_url). It then resolves the hostname's DNS and refuses to connect if that resolves to a private, loopback, link-local, multicast, reserved, or unspecified address, which catches a public-looking hostname that actually points inward (#resolved_url_is_public, api/services/llms_txt/fetch.py#_is_public). Both checks run again on every redirect hop, not just the first request, and the fetch gives up after 3 redirects and caps what it reads at 512,000 bytes (#fetch_llms_txt, #_MAX_REDIRECTS, #LLMS_TXT_MAX_BYTES). This guards this one fetch; it is not a general claim about the product's security posture.

Rule-derived, and model-written

The checks above are plain code with no model involved: the same input always produces the same checklist. What is written afterward, the summary, the suggested fixes, and the proposed replacement file, is a model's prose and draft text, built from your project's identity and the checklist results and framed to it as data, not instructions (api/services/llms_txt/service.py#build_review_prompt, #REVIEW_SCHEMA). The fetch and the checklist are saved before that model call runs, so if the model phase fails, the fetch result and the checklist still show on screen; only the written summary, suggestions, and proposed file are missing, and running the check again retries the whole thing (#run_llms_txt_audit).

What's on the screen

The headline reads Compliant, Not compliant, No llms.txt, or Not reachable, depending on whether the site answered, whether it served the file, and what the checklist found (frontend/src/routes/(app)/llms-txt-advisor/+page.svelte#compliant). Below it, every check that ran is listed with its level; the warning and failure counts shown are drawn from that same list, not written separately (#checks, #failCount, #warnCount). A check that got no answer from the site lists no checks at all, since none ran. If the checks cannot be loaded, the screen shows an error with Retry rather than "No check run yet". This documentation site is generated the same way discoveredby.ai's own /llms.txt is, both built from a single index rather than hand maintained, though this site's own file is not run through the advisor.

  • Weekly report: the other periodic artifact on the account, with no model involved anywhere in it
  • Growth Advisor: the other advisor whose output is model-written rather than derived
  • What is GEO?: the broader idea of making a site legible to AI systems that this file is one piece of

Last verified 2026-10-14

Start monitoring your AI visibility.

See how AI search engines talk about your brand.

Free to start. No credit card required.