All documentation

Optimize

Site audit

Crawl your own site as DiscoveredByBot, check every page against 238 technical SEO rules in 17 sections, and review issue priorities and page reports.

Open Site health → Site audit to crawl your own site as DiscoveredByBot and check every page it finds against 238 technical SEO rules in 17 sections. Each audit gets a health score, a prioritized list of issues to review and a report for every URL it fetched.

The site audit's overview for the fictional quillstone.io: a health score of 82.7 out of 100 marked Needs work, 36 open issues across 12 pages with 4 high priority, Social media metadata as the weakest area at 21.4, 70.1% evidence coverage, the first six issues to fix, and each section's health, weakest first

What a site audit crawls

An audit starts at your project's URL (api/modules/website_crawler/projects.py#start_crawl) and queues more URLs from that same host as it goes:

  • the links on each page it loads;
  • the canonical and language-alternate (hreflang) URLs a page declares;
  • the pages your sitemaps list, read once from the start page: the Sitemap: lines in robots.txt, or /sitemap.xml when there are none, following sitemap index files, up to 10 sitemap files and 50,000 entries (api/modules/website_crawler/sitemaps.py#capture_sitemaps, #MAX_DOCUMENTS, #MAX_URLS).

If the start page redirects to another host, for example from the apex domain to its www subdomain, the audit follows that final host instead. URLs on any other host, a different subdomain included, are not crawled (api/modules/website_crawler/service.py#_save_fetched_url).

Every URL is requested as DiscoveredByBot/1.0 (+https://discoveredby.ai) (api/modules/website_crawler/robots.py#CRAWLER_USER_AGENT). Before each fetch the audit reads robots.txt again for that URL (#is_allowed_by_robots), then loads the page in a browser, three pages at a time, with up to 90 seconds for each page (api/modules/website_crawler/service.py#fetch_website, #CRAWL_CONCURRENCY). Each URL ends in one of three states:

  • Successful: the URL answered with an HTTP 2xx status. An HTML page is analysed; any other content type is recorded without page analysis.
  • Robots blocked: robots.txt disallows the URL for DiscoveredByBot, so it is not fetched. A robots.txt that is missing outright (an HTTP 400 to 499 response other than 429) places no restriction.
  • Failed: the URL returned an error status, timed out, returned no HTML or redirected to a non-public address, or its robots.txt could not be read, in which case the page is not fetched. robots.txt counts as unreadable on a server error, a network failure or a 429 rate limit, and when the file served there is HTML or is not UTF-8 text (api/services/robots/fetch.py#fetch_robots).

Besides the pages themselves, an audit sends a bounded number of HEAD requests to check the status of things pages point to, such as images, language alternates and links to other sites, without crawling them: at most 100 per audit and 4 new ones per page. Before each one it fetches robots.txt from that URL's host, other sites included, and skips the request when DiscoveredByBot is disallowed or the file cannot be read (api/modules/website_crawler/http_checks.py#MAX_CHECKS, #MAX_PER_PAGE, #probe_url). It also downloads up to 50 social preview images per audit, which may be hosted on other sites, after the same robots.txt check on each image's host (api/modules/website_crawler/social_images.py#MAX_SESSION, #fetch_social_image).

Start and stop an audit

Company members and editors can start an audit when the company's plan includes robots diagnostics: the standard Starter, Growth and Pro plans include it, Free and Trial do not, and custom plans follow their saved feature setting (api/modules/website_crawler/projects.py#start_crawl, #ENTITLEMENT_REASON, api/services/entitlements.py#standard_entitlements). Every current project member, viewers included, can read every saved audit (api/modules/website_crawler/access.py#authorize_website).

Choose how many URLs to crawl, from 1 to 1,000, 100 by default (api/modules/website_crawler/schemas.py#ProjectCrawlStart). The number counts every URL the audit queues, the start page included, whether it turns out to be an HTML page, a file or a failure. When the audit finds more URLs than it has room for, it stops queueing new ones and shows a note that it reached its limit (api/modules/website_crawler/service.py#_within_page_limit).

One audit runs per project at a time, and a new one can start at most once a minute (api/modules/website_crawler/projects.py#start_crawl, #START_COOLDOWN). An audit that has made no progress for 30 minutes no longer blocks a new start: the new audit replaces it, and the old one is marked failed (api/modules/website_crawler/service.py#can_queue, #CRAWL_LEASE, #prepare_crawl). When a batch of pages hits an unexpected error, its pages go back in the queue and the batch is retried a minute later. Pages whose batch still fails after three retries in a row are marked failed with "This page could not be crawled after several attempts.", and the audit carries on without them. The audit itself fails, with "The audit stopped unexpectedly before it finished. Start a new audit.", only when the step that keeps failing is not fetching pages, such as building the audit's report (api/modules/website_crawler/tasks.py#crawl_website_task, #_release_claims, #GAVE_UP_ON_PAGE, #STOPPED_UNEXPECTEDLY). Starting an audit removes the oldest audits beyond the newest 10, with their results (api/modules/website_crawler/projects.py#RETAIN_SESSIONS).

While an audit runs, the screen updates every few seconds (frontend/src/lib/components/crawler/CrawlerRefresh.svelte#refresh) and its views show partial results from the pages fetched so far. After the last page is saved, one more step builds the section reports, the score and each page's issue count, and the audit then reads Completed (api/modules/website_crawler/service.py#finish_session). Each page is loaded in a browser, so a large audit can take hours.

Company members and editors can stop a running audit, and stopping never checks the plan (api/modules/website_crawler/projects.py#stop). Pages already being fetched finish and are saved; URLs still waiting are dropped, and nothing new is discovered. The audit then completes with what it fetched, and shows a note that it stopped early. If nothing had been fetched and nothing was being fetched when you stopped it, the audit fails with "Stopped before any page was fetched." instead (api/modules/website_crawler/service.py#stop_crawl, #STOPPED_EMPTY, #_save_url_result).

Read the results

Choose an audit from the Showing list at the top of the screen. Each audit has six views. Results from the earlier version of the site audit are not carried over.

Overview opens on the audit's verdict:

  • SEO health score: for each rule with measured outcomes, its passing outcomes divided by its passing plus issue outcomes, averaged across those rules. The 13 informational rules are not scored, and every other rule counts equally. The score's ring is green from 90, amber from 50 and red below 50. A score that would round to 100 while any measured rule has an issue shows 99.9, for the site and for each section (api/modules/website_crawler/audit.py#_metrics).
  • Open issues: how many issues are open, how many of them are high, medium and low priority (each count opens Issue priorities filtered to it), and the weakest area, the lowest-scoring section with an issue. An issue marked accepted risk or fixed is no longer open.
  • Evidence coverage: the average share of each applicable rule's outcomes that were fully measured. Unknown and pending outcomes never count as passes.

While the audit runs, the score is marked Partial and its ring is grey. A finished audit with a failed or blocked URL, or with unknown outcomes for any rule, says that some checks could not be measured. The score covers measured evidence only and does not predict search rankings or certify accessibility (api/modules/website_crawler/audit.py#build_audit_dashboard, #dashboard_metadata, #WEIGHTS, api/modules/website_crawler/analysis.py#SEO_RULES).

Below the verdict, Fix these first lists the first 6 open issues in the Issue priorities order, each with its recommendation, section and estimated effort and how many of the measured URLs it affects, and links to the issue's rule report (api/modules/website_crawler/prioritization.py#overview_priorities). Beside it, Health by area lists the sections that have issues, weakest first, with each one's score and number of issues, and marks a section with under 70% of its outcomes fully measured; the sections without issues share one line, where a thinly measured one keeps its measured share. A line below both counts the audit's URLs (discovered, successful, failed and blocked by robots.txt) and says when the audit reached its URL limit or was stopped early.

Issue priorities has one item for each rule that flagged at least one outcome, informational rules excluded, with a suggested priority. Its score adds the rule's impact (75 for high, 45 for medium, 15 for low), up to 15 for the share of measured outcomes affected and up to 9 for how many are affected; 70 and above is high, 40 and above medium. It is technical triage, not a prediction of rankings or business impact (api/modules/website_crawler/prioritization.py#prioritize_rule, #build_priority_queue). The issues are listed highest priority first, each with the same row as Fix these first. Filter them by priority, where each choice counts the issues the other filters leave, or by section, review decision (open or in progress by default) and severity, or search them. How priority works, at the end of the list, has the scoring in full; the suggested score of each issue is on its rule report.

Company members and editors record a decision on each issue from its rule report: Open, In progress, Accepted risk or Fixed, awaiting recrawl, with an optional priority override and a note. A note is required for an override, an accepted risk or a claimed fix (api/modules/website_crawler/schemas.py#IssueReviewUpdate, api/modules/website_crawler/router.py#update_issue_review). Decisions can be saved only once the audit has finished. A decision applies to the whole rule in that audit only and never changes its findings or its score, so verify a fix with a new audit. Viewers see the same guidance without the decision form.

Site report lists all 238 rules, grouped by section, each section with its score and how many of its rules need review. Filter the rules by outcome: Needs review, Incomplete (some outcomes unknown or pending, which never count as passing), Passed, Informational or Not applicable, each choice counting the rules the search, section and severity filters leave. An informational rule sits under Informational whatever it recorded, because informational rules are not scored and are never issues (frontend/src/lib/website-crawler.js#ruleBucket). Review decisions do not move a rule: one marked accepted risk or fixed still needs review here until a new audit measures it again, so after decisions this count can be higher than the open issues on Overview and Issue priorities. With every outcome shown, a section lists its rules needing review in full, with their recommendation and how many measured URLs they affect, and folds the rest into one line that says what they are. A rule opens its rule report, which lists every URL's outcome for that rule with its evidence, plus how to investigate and verify it (api/modules/website_crawler/reporting.py#rule_results, api/modules/website_crawler/prioritization.py#rule_guidance).

URLs lists every URL the audit queued with its status, HTTP status, response time and crawl time; search it or filter it by status.

Technical SEO gives each of the 17 sections its own summary: page titles, meta descriptions, headings, content quality, canonical URLs, HTTP status and redirects, crawlability and indexability, XML sitemaps, internal linking and architecture, images and media, performance and Core Web Vitals, structured data and schema, international SEO, JavaScript SEO, mobile SEO and accessibility, URL structure and technical hygiene, and social media metadata (api/modules/website_crawler/audit.py#SECTIONS). The list of sections shows each one's score, coloured like Health by area when it has issues, or that it has none. A section opens with its score and how many of its checks have issues; its checks with issues come first, each opening to its recommendation, example URLs and a link to all its results, and the rest fold into one line. What the section's evidence can and cannot establish is under About this section's evidence.

Lighthouse shows lab results only when the crawl could run Lighthouse for a page; otherwise the page says why it has none, for example that Lighthouse could not run. Where they exist, they cover mobile and desktop: Performance, Accessibility, Best Practices, SEO and the experimental Agentic Browsing. Lab results are not real-user Core Web Vitals (api/modules/website_crawler/lighthouse.py#capture_lighthouse).

Read one page

The URL report for quillstone.io/pricing from the fictional demo's audit: HTTP status 200, 98 words, not blocked by robots.txt, and 238 checks, 19 needing review (4 of them partial findings), 47 incomplete, 108 passed, 13 informational and 51 not applicable, filtered to the checks that need review and grouped by section

Choose any URL in an audit to open its URL report, at an address under /site-audit/urls/. Its header shows the HTTP status, response time, crawl time, word count, whether robots.txt blocked it and its final URL. Three tabs follow:

  • URL report: all 238 checks for this URL, grouped and filtered like the site report, each Needs review, Passed, Unknown or pending, not applicable or informational, with a link to its evidence. An informational rule that flagged this URL reads Informational and sits under that filter, because informational rules are not scored and the page issue queue does not count them. A confirmed issue with incomplete evidence is a partial finding and still counts as needing review; the report says how many there are. Sitemap-file checks are marked as site checks and belong to the site report (api/modules/website_crawler/reporting.py#build_url_report).
  • Technical SEO: each section's saved evidence for this page.
  • Lighthouse: only when the crawl could run Lighthouse for this page, its lab results; otherwise why it has none.

Raw HTML versus rendered

Crawlers that do not run JavaScript only ever see the raw HTML a page delivers. Content, links or metadata that appear only after scripts run are invisible to them, even though a person opening the page in a browser sees them. The JavaScript SEO section compares each page's raw HTML response with the same page after the browser ran its scripts: whether rendering failed, whether important content or links differ between the two, whether the title, meta description, canonical or robots directives change, client-side soft 404s, blocked JavaScript resources, click-only navigation, rendering time and console errors (api/modules/website_crawler/javascript.py#build_javascript).

How it feeds the page issue queue

When an audit completes, each URL gets a count of the rules whose outcome for it is an issue. Informational rules and unknown outcomes are not counted (api/modules/website_crawler/reporting.py#refresh_url_issue_counts).

The Page issue queue reads only the project's newest completed audit, so a running or failed audit never hides the previous completed one's results; a stopped audit that completed counts (api/services/page_issues.py#inventory_statement). Each URL that audit fetched, could not fetch or found blocked appears there as one Site audit item linking to its URL report:

  • Needs action when any SEO rule other than an informational one flags the page, with the number of rules that do;
  • Diagnostics unknown when no rule flags it, because rules the audit could not assess are not passes;
  • Diagnostics unknown, with the reason, when the audit could not read the page or robots.txt blocked it (api/services/page_issues.py#page_issues).

Free site audit

Anyone can run a free audit of any public site at discoveredby.ai/site-audit, without an account or a project. Only our website's server can start one: the audit's API refuses any call without its shared secret, and each start needs a passed person check (Cloudflare Turnstile) and stays within the free audit's limits (api/modules/free_audit/router.py#start, api/modules/free_audit/gate.py#public_caller, #verify_turnstile, api/modules/free_audit/service.py#start_audit). It uses this same crawler, requesting pages as DiscoveredByBot and reading robots.txt the same way, and checks the same 238 rules in 17 sections, starting at the home page of the site entered. It crawls up to 25 pages. A visitor who verifies an email address with a one-time code can upgrade it, once, to a new crawl of up to 100 pages (api/modules/free_audit/models.py#FREE_PAGES, #VERIFIED_PAGES, api/modules/free_audit/service.py#queue_upgrade), and each verified address receives the report by email. A site holds one free audit at a time, and anyone who enters that site sees its results (api/modules/free_audit/service.py#held_audit); the results page asks search engines not to index it. The crawl data is kept while a crawl waits or runs, then deleted 48 hours after the crawl whose results the audit shows finishes, or at once when the audit's first crawl saves no page (api/modules/free_audit/tick.py#HOLD, #_complete, #_fail, #prune_expired). An upgrade crawl that saves no page is deleted at once, and the audit keeps its 25-page results until their own 48 hours run out (api/modules/free_audit/tick.py#_fall_back). A new free audit of the site can start once the data is deleted. A free audit belongs to no project, so it never feeds the page issue queue, and nobody can record issue decisions on it.

Safety

Every browser load, including every resource the page loads, goes through a forward proxy that only allows the public internet on ports 80 and 443. A request to a private, loopback or link-local address is refused rather than followed (api/services/browser/egress.py#ALLOWED_PORTS, #public_addresses). Each page's browser render is bounded to 200 proxy connections and 15,000,000 downloaded bytes (#MAX_CONNECTIONS, #MAX_UPSTREAM_BYTES); when the crawl can run Lighthouse, its separate load of the page goes through its own proxy, bounded to 400 connections and 30,000,000 bytes (api/modules/website_crawler/lighthouse.py#capture_lighthouse). The browser also disables WebRTC's non-proxied path, so a page cannot use it to reach the network outside the proxy (api/services/browser/render.py#render). Requests for robots.txt, sitemaps, the HEAD checks and social preview images go only to public addresses too (api/services/robots/fetch.py#fetch_robots).

Page scripts run during the render, and requests to common analytics and tag-manager services (for example Google Analytics, Google Tag Manager, Meta Pixel, Hotjar, Microsoft Clarity and Segment) are blocked, so an audit does not add page views to those tools. First-party or self-hosted analytics, for example a server-side tag manager or analytics proxied through your own domain, are not blocked (api/services/browser/render.py#ANALYTICS_HOSTS, api/services/browser/egress.py#host_matches).

What a site audit does not do

  • It requests pages as DiscoveredByBot only, and follows robots.txt as DiscoveredByBot. Its saved robots evidence also reads your robots.txt for Googlebot, and in the image and social sections for Googlebot-Image, Twitterbot and facebookexternalhit (api/modules/website_crawler/robots_analysis.py#BOTS, api/modules/website_crawler/images_media.py#IMAGE_BOTS, api/modules/website_crawler/social_images.py#BOTS). It does not act as any other crawler, and it does not check each AI crawler's robots.txt access page by page; Robots diagnostics checks the declared policy for five AI crawlers.
  • It does not see real bot traffic. AI crawlers reads your own server or CDN logs for that.
  • It does not run the Saved page diagnostics checks on each page it crawls; run those on a page you choose.
  • It does not establish complete coverage of your site: a page nothing links to and no sitemap lists is not found, and an audit stops queueing at the number of URLs you chose.
  • DiscoveredBy does not edit your site.
  • Page issue queue: your newest completed audit's pages, beside saved gaps, optimizations and diagnostics.
  • Saved page diagnostics: check one exact URL on demand and keep its recheck history.
  • Robots diagnostics: the declared robots.txt policy for five AI crawlers on one URL.
  • AI crawlers: which AI crawlers actually visited your pages, read from your own server logs.
  • Plans and limits: feature access follows the company's plan.

Last verified 2026-10-19

Start monitoring your AI visibility.

See how AI search engines talk about your brand.

Free to start. No credit card required.