llms.txt and AI Crawler Access: The 20-Minute Setup

Google Search's own documentation says you don't need to create an llms.txt file. Chrome's Lighthouse now runs an audit that checks whether you have one — and fails it if your links aren't in Markdown syntax. Same company, two answers, and a lot of teams spending their technical GEO time on the wrong file.

Here's the honest version of that setup, ordered by evidence rather than novelty. The 20 minutes that decide whether ChatGPT, Gemini, Claude, and Perplexity can see your pages at all are an access audit. llms.txt is the last five minutes — worth shipping, cheap to get right, and not the thing standing between you and a citation.

The contradiction at the top of Google

Google Search Central's guidance on AI features is unambiguous: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." The same page, last updated December 10, 2025, says a page "must be indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode, and that there's no special schema.org structured data you need to add either. The one technical instruction it does give: ensure "that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure."

Meanwhile, over in Chrome, Lighthouse's documentation describes llms.txt as "an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents." Lighthouse's Agentic Browsing category contains an llms-txt audit alongside five others — an agent accessibility tree check, cumulative layout shift, and three WebMCP audits. The audit is marked Not Applicable if the file 404s, "as providing the file is optional at the moment," but it fails if the request returns a server error. That doc was last updated May 5, 2026.

Search Engine Journal flagged the split in May 2026, and the right read is not that one team is wrong. They're answering different questions. Google Search is answering "does this file affect whether you rank and get pulled into an AI Overview?" — no. Chrome is answering "is this site legible to a browsing agent?" — a different bar, on a different timeline.

The practical consequence showed up on July 3, 2026, when Slobodan Manic ran the Agentic Browsing audit on nohacks.co and got the message "File does not appear to contain any links" — despite every link in the file working perfectly. Lighthouse parses llms.txt as Markdown and only counts [text](url) syntax. Rewriting entries like - Homepage: / - Publication masthead as - [Homepage](/): Publication masthead moved the category score from 0.67 to 1.0. About five characters per link.

The audit's documented failure conditions are worth memorizing because they take a minute to satisfy: no H1, no Markdown-formatted links, or content under 50 characters ("suspiciously short").

What llms.txt actually is

The llms.txt proposal is exactly that — a proposal, not a standard adopted by any search or AI vendor. Its stated purpose is to "provide information to help LLMs use a website at inference time," and it lives at the root path /llms.txt. Note the phrase inference time: this was designed for the context-window problem, not for training and not for ranking.

The required structure is small:

• An H1 with the project or site name. The spec is explicit that "this is the only required section."
• A blockquote summarizing the site.
• Zero or more Markdown sections with no headings, for extra detail.
• Zero or more H2-delimited file lists of [name](url) links, each with an optional : note.

One convention is easy to miss: a section literally titled "Optional" has special meaning — "the URLs provided there can be skipped if a shorter context is needed."

For a well-formed example, look at Cloudflare's developer docs llms.txt. It opens with an H1, then a single blockquote explaining that each product links to its own llms.txt, then H2 sections of annotated Markdown links. It's an index of indexes, running roughly 1,500–1,700 lines. Anthropic publishes one for its platform docs too, alongside an llms-full.txt.

And here is where the convention gets strange. A community directory lists roughly 849 sites with the token size of each file: Anthropic's Claude file is about 892 tokens, Cloudflare's about 49K, the Vercel AI SDK's about 293K. Same convention, three orders of magnitude apart. In practice "llms.txt" names two different artifacts — a map of the site and a dump of it — and the dump defeats the context-window purpose the spec was written for. Ship the map.

What the data says about llms.txt and citations

SE Ranking studied 300,000 domains and published the results in November 2025. Just 10.13% had an llms.txt file. More importantly, the study — which combined XGBoost regression, Spearman correlation, and SHAP values — found no correlation between AI citations and llms.txt, and the researchers noted that "removing this variable from our XGBoost model actually improved its accuracy." On crawler behavior, they observed that some SEO logs show GPTBot occasionally fetching llms.txt files, "though this doesn't happen often."

Search Engine Journal's coverage of the study added the right caveat: the file "doesn't seem to directly impact AI citation frequency. At least not yet."

Google's John Mueller made the skeptical case earlier, in a Reddit thread reported by SEJ in April 2025: "To me, it's comparable to the keywords meta tag – this is what a site-owner claims their site is about." And the follow-up that matters most: "Is the site really like that? well, you can check it. At that point, why not just check the site directly?"

That objection is the single best reason to keep llms.txt short and accurate: a file that drifts from your live pages is a self-description nobody has reason to trust.

There's a real counter-case, though. Wix's AI Search Lab reviewed 1,400+ llms.txt files starting in November 2025 and revisited them six months later. Their July 2026 write-up reports over 125,000 llms.txt files indexed in Google Search, with the files themselves ranking in Google, surfacing in AI Mode, and appearing in ChatGPT and Perplexity answers. That's a discovery path that has nothing to do with any crawler formally "supporting" the convention — it's just a well-structured crawlable asset.

Adoption is also partly a plugin artifact. Yoast ships automatic llms.txt generation to all users for free: "The file is generated and refreshed automatically, no coding or manual work needed." A meaningful share of that 10.13% is a toggle, not a decision.

Fair verdict: llms.txt is cheap, harmless, unproven as a citation lever, and now audited by a browser. Ship a good one in five minutes. Don't budget more than five minutes for it, and don't expect it to move your visibility on its own.

One caution on the claims you'll read elsewhere: several 2026 posts assert that Anthropic or Perplexity "support" llms.txt. No primary vendor documentation says so — not OpenAI's crawler docs, not Anthropic's crawler support article, not Perplexity's bots doc. Treat vendor-support claims as unsourced until someone shows you the primary page.

The part that actually decides whether engines can see you

Crawler access isn't emerging or unproven. It's binary, and it's routinely broken by settings nobody consciously chose.

Start by separating bots by purpose, not by vendor. Training crawlers collect content that may be used to train or fine-tune models. Retrieval crawlers build the index an engine consults when answering a live question — these decide whether you can be cited. User-initiated fetchers grab a page because a human just asked about it. Here's what the vendors document:

Operator	Bots	What each is for
OpenAI	GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot	training; ChatGPT search; user-initiated fetches; ads validation
Anthropic	ClaudeBot, Claude-SearchBot, Claude-User	training; search quality; user-initiated
Perplexity	PerplexityBot, Perplexity-User	search indexing; user-initiated
Meta	meta-externalagent, meta-externalfetcher, facebookexternalhit	training/indexing; user-requested fetches; integrity checks
Google	Googlebot, Google-Extended	Search (and therefore AI features); Gemini training and grounding

A few details from the primary docs are worth having in front of you before you write a single Disallow line.

OpenAI now documents four bots, including OAI-AdsBot, which is "used to validate the safety of web pages submitted as ads on ChatGPT" and whose data "isn't used for model training." Many checklists still list only three. OpenAI also states the cost of blocking retrieval plainly: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." And on user-initiated traffic: "Because these actions are initiated by a user, robots.txt rules may not apply."

Anthropic runs three bots — ClaudeBot for content "that could potentially contribute to their training," Claude-User for user-initiated requests, and Claude-SearchBot, which "navigates the web to improve search result quality for users." Anthropic supports the Crawl-delay extension and, critically, warns that robots.txt rules must be applied to every subdomain you want covered.

Perplexity documents two: PerplexityBot, which "is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models," and Perplexity-User, of which the docs say "this fetcher generally ignores robots.txt rules."

Meta documents three, including meta-externalfetcher, which "may bypass robots.txt because it performs fetches that were requested by the user."

And Google-Extended is not a search lever. It governs whether crawled content may be used to train future Gemini models and for grounding in Gemini apps. Google states outright that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Blocking it to stay out of AI Overviews doesn't work, because AI Overviews run on ordinary Search indexing.

The volume picture explains why blanket blocking is such a poor trade. Over twelve months to mid-2025, Cloudflare found training accounted for 80% of AI crawling, search about 18%, and "just 2% for user actions" — 82/15/3 in the most recent six months of that window. The 15–18% slice is the part that decides whether you get cited. Most one-click "block AI bots" switches can't tell it apart from the 80%.

Grok is the genuine gap here. xAI's web-search documentation describes a tool that "enables Grok to search the web in real-time and browse web pages," with allowed_domains and excluded_domains parameters capped at five each — but it names no crawler user agent and publishes no IP list. For Grok, technical GEO is mostly measurement and off-site presence rather than configuration.

Defaults you didn't choose

Cloudflare flipped its default to blocking AI crawlers on July 1, 2025, framing it with a stat worth sitting with: relative to a decade ago, it's roughly 750× harder to get a visitor from OpenAI and 30,000× harder from Anthropic than from Google. On July 1, 2026 the control got more granular — AI traffic split into Search, Agent, and Training as three separately controllable categories — with a deadline attached: September 15, 2026, after which Training and Agent are blocked by default on ad-monetized pages for newly onboarded domains. Search stays allowed by default. (We work through that decision bot by bot in Cloudflare's Sept 15 AI crawler rules.)

Cloudflare also had managed robots.txt enabled on "over 3.8 million domains" as of September 24, 2025. Millions of robots.txt files are maintained by a platform rather than by the person responsible for the site's visibility.

What that looks like when it goes wrong: a Brooklyn law firm ran an AI-visibility audit on July 3, 2026 and found GPTBot, PerplexityBot, ClaudeBot, and Common Crawl all receiving 403s from Cloudflare, dating back to November 2025. In the 24 hours before the fix: GPTBot 0 allowed, 515 blocked; PerplexityBot 0 allowed, 288 blocked. The cause was two settings that shipped as defaults when the zone was created — AI crawler blocking in the Security tab, plus an AI opt-out signal injected into robots.txt. After flipping both toggles, GPTBot went from 0 to 14,292 requests on day one and PerplexityBot from 0 to 7,483. (That account is the firm's own marketing article; the crawler counts are specific and credible, but the revenue figures it also reports have no control group.)

Seven months of invisibility, zero errors in any dashboard, because nothing failed. The pages were fine. The door was closed.

A few robots.txt mechanics from Google's spec that produce the same silent class of failure:

• Rules apply "only to the host, protocol, and port number where the robots.txt file is hosted." Your apex domain, www, docs., and blog. each need their own file.
• Google "generally caches the contents of robots.txt file for up to 24 hours."
• Google "enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored."
• Field names are case-insensitive; path values are case-sensitive.
• Google's crawlers "treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist," while a 5xx stops crawling for around 12 hours before falling back to a cached copy for up to 30 days. A robots.txt that 500s is a crawl-stopping bug.
• crawl-delay is not supported by Google, even though Anthropic documents it.

Above robots.txt sits a preferences layer. Cloudflare's Content Signals Policy, launched September 24, 2025, defines search, ai-input, and ai-train, expressed like this:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

The July 2026 update adds a use= level — immediate, reference (the default: index, excerpt, link back), or full. Cloudflare's own caveat deserves quoting: "content signals express preferences; they are not technical countermeasures against scraping." That user-initiated fetchers may ignore robots.txt is the proof of the limit.

The invisible-content problem

The second silent failure has nothing to do with permissions. Vercel and MERJ monitored nextjs.org and Vercel network traffic and published the finding in December 2024: "none of the major AI crawlers currently render JavaScript" — covering OpenAI, Anthropic, Meta, ByteDance, and Perplexity bots. They fetch JavaScript files without executing them (11.50% of ChatGPT's requests, 23.84% of Claude's). Read that with its date attached: no vendor has documented a rendering AI crawler since, but the honest framing in mid-2026 is "the last definitive measurement said no," not "this is permanent."

Gemini is the exception, for a structural reason: Google's AI features depend on ordinary Search indexing and snippet eligibility, which means Googlebot's rendering pipeline, not a separate AI crawler.

The same study found AI crawlers are wasteful in ways that punish sloppy URL hygiene: 34.82% of ChatGPT's requests and 34.16% of Claude's hit 404s, against 8.22% for Googlebot. ChatGPT spent another 14.36% of its fetches following redirects. For scale, in the month before December 17, 2024, GPTBot made 569M fetches and Claude 370M — the AI crawlers combined reached a little over 28% of Googlebot's 4.5B.

The two-minute test: open view-source on your most important page — not DevTools' rendered DOM, which shows you the page after JavaScript has run. If the sentence you want quoted in an AI answer isn't in that raw HTML, most AI crawlers never see it.

The 20-minute checklist, in order

Minutes 0–5 — prove the retrieval bots get a 200.

1. Request your top pages as each documented retrieval agent (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) plus the user-initiated ones (ChatGPT-User, Claude-User), and record the status code. A 403, a 503, or a CDN challenge page is the failure mode above. Geoptimizer's free AI Crawler Checker does exactly this for GPTBot, ClaudeBot, and eight other AI crawlers with no signup, if you'd rather not hand-roll ten curl commands.
2. Check every hostname separately — apex, www, docs., blog..
3. Confirm robots.txt itself returns 200, not a 5xx.

Minutes 5–10 — decide training vs retrieval deliberately.

1. Write the decision down per category, not per vendor: Training (GPTBot, ClaudeBot, meta-externalagent, CCBot, Google-Extended) vs Retrieval (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) vs user-initiated fetchers.
2. Price each choice against the vendor docs. Disallowing OAI-SearchBot removes you from ChatGPT search answers. Disallowing Google-Extended costs you nothing in Search inclusion or ranking. Disallowing user-initiated fetchers largely doesn't work, because OpenAI, Perplexity, and Meta all document that those requests may ignore robots.txt.
3. Optionally add a Content-Signal line alongside your rules to state the preference even where you're not enforcing it.

Minutes 10–15 — make the page readable without JavaScript.

1. View-source your money pages. If the answer text isn't there, fix that before anything else on this list.
2. Clean up 404s and redirect chains.
3. Keep nosnippet, data-nosnippet, max-snippet, and noindex off your key pages — those are precisely the controls Google names for reducing AI-feature exposure.

Minutes 15–20 — ship llms.txt correctly.

1. H1, blockquote summary, H2 sections of [name](url): note links, served at /llms.txt, over 50 characters, links in Markdown syntax. That satisfies both the spec and what Lighthouse validates.
2. Keep it small and keep it true to the live site.
3. Expect roughly 24 hours, not instant results. OpenAI notes "it can take ~24 hours from a site's robots.txt update for our systems to adjust"; Perplexity says changes may take up to 24 hours; Google caches robots.txt for up to 24 hours.

Then verify by IP, not by user agent string — user agents are trivially spoofable. OpenAI publishes per-bot IP ranges (gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json), Anthropic publishes claude.com/crawling/bots.json, and Perplexity publishes its own. Longer term, cryptographic identity replaces IP lists too: Cloudflare's signed agents classification, launched August 28, 2025, uses HTTP message signatures so an agent can prove who it is rather than claim it.

If you'd rather see all of this in one pass, the free GEO Site Audit checks AI crawler access, llms.txt, structured data, and content shape together — the Free plan includes five audits a month and doesn't expire.

What to re-check, and how to know it worked

Three things are moving and worth a quarterly look:

• IETF AIPREF, the standards-track attempt to replace vendor-specific syntax. Draft-ietf-aipref-vocab-06, dated April 28, 2026, defines two categories — AI model training and Search — with values allow (y), disallow (n), or unknown, expressible as a Content-Usage header or a robots.txt rule. It is on the Proposed Standard track and not yet an RFC.
• Cloudflare's September 15, 2026 defaults, which are reason enough to re-run your access check in late September even if it passed in July. We keep the ongoing coverage of crawler-policy changes on the Geoptimizer blog.
• WebMCP, a proposed web standard for exposing structured tools to AI agents, in origin trial from Chrome 149. Three of Lighthouse's six Agentic Browsing audits already point at it.

Now the honest close. Technical access is necessary, not sufficient. The strongest published correlate of AI-answer brand visibility isn't anything on your server: Ahrefs studied 75,000 brands and found branded web mentions correlated with AI Overview brand mentions at 0.664 — far above Domain Rating (0.326) or backlinks (0.218). Brands in the top quartile for web mentions averaged 169 AI Overview mentions against 14 in the next quartile. The authors stress that correlation isn't causation, and the same caution applies here: 20 minutes of robots.txt work removes a blocker, it doesn't manufacture authority.

Which raises the question this whole exercise depends on: how do you know the fix worked? You can't watch ChatGPT's index. You can watch the answers. That's what Geoptimizer measures — your buyers' real questions run live on ChatGPT, Gemini, Claude, and Grok, turned into a 0–100 AI Visibility Score built from mention rate, citation rate, prominence, and sentiment, with the weights published rather than hidden. Because AI answers are nondeterministic, the headline score uses a 7-day rolling window with a confidence band; single scans are labeled as snapshots.

The useful workflow: run a scan before you touch anything, make the access fixes, wait out the ~24-hour propagation window, then scan again. It also happens to be the only feedback loop available for Grok, where there's no crawler contract to configure.

Start with the free AI Crawler Checker and GEO Site Audit — no signup, no credit card — and see which of the ten crawlers can actually read your site right now.

FAQ

Do I need an llms.txt file to appear in AI answers? No. Google Search's documentation states you don't need new machine-readable files or AI text files to appear in its AI features, and a 300,000-domain study found no correlation between llms.txt and AI citations. It's still worth five minutes: Chrome's Lighthouse audits for it, it's a crawlable asset, and Wix's research found 125,000+ such files indexed in Google.

Which AI crawler should I never block? The retrieval ones, if you want to be cited: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot. OpenAI states directly that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." Blocking training crawlers like GPTBot and ClaudeBot is a separate, defensible decision.

Does blocking Google-Extended keep me out of AI Overviews? No. Google's documentation says Google-Extended governs Gemini model training and grounding, and "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews draw on ordinary Search indexing, so snippet controls like nosnippet and max-snippet are the relevant levers there.

If you want tactical, step‑by‑step guidance for opting out of AI Overviews without sacrificing search visibility, see how to opt out of AI overviews without losing rankings.

How long until a robots.txt change takes effect? Plan on about 24 hours. OpenAI documents roughly 24 hours for its search systems to adjust, Perplexity says up to 24 hours, and Google caches robots.txt for up to 24 hours. Re-check your access, then re-measure your visibility after that window rather than the same afternoon.