Cloudflare's Sept 15 AI Crawler Rules: Who to Let In

A law firm spent seven months invisible to ChatGPT and Perplexity and never saw it in a single report. Its site had moved onto Cloudflare in November 2025 with AI crawler blocking switched on by default, and on July 3, 2026 a visibility audit finally surfaced the logs: GPTBot had made 0 allowed requests and 515 blocked in the previous 24 hours; PerplexityBot, 0 allowed and 288 blocked. The firm's own summary of why nobody noticed is the best sentence written about this whole problem: "You can't see traffic that never arrives."

On September 15, 2026, Cloudflare's defaults change again — and Cloudflare says it sits in front of more than 20% of web domains. Here's what actually changes, the one checkbox that can take you out of Google Search, and a bot-by-bot framework for deciding which AI crawlers to keep letting in.

What Cloudflare actually announced (and what it didn't)

On July 1, 2026, Cloudflare replaced its binary "AI bot / not AI bot" controls with behaviour categories. Instead of one switch, you now govern crawlers by what they're doing. The three that dominate the conversation:

• Search — collecting and indexing content to answer questions later and surface links.
• Agent — automated behaviour acting in real time on a person's behalf.
• Training — taking content to train or fine-tune a model.

The published list runs further — Transact, Data Collection, Security Testing, SEO, Ads Verification, Social/Link Preview, Feed Fetching, Monitoring & Operations — but those three carry the visibility consequences.

Here is the September 15 default in Cloudflare's own words: "For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default."

Read that twice, because the widely repeated summary — "Cloudflare will block AI crawlers by default" — is materially broader than what Cloudflare wrote. Three qualifiers do a lot of work:

1. It's Training and Agent, not Search. The category most closely tied to whether an engine can cite you stays open by default.
2. It applies to pages that display ads. Cloudflare's rationale: "An ad is a signal that a website owner meant for a person to land there and see it — something monetizable that fuels the business." Cloudflare's earlier documentation of the same mechanism explains that ad detection operates at hostname level. A B2B SaaS marketing site with no ad units doesn't trip the trigger.
3. It's for new domains onboarding. Press coverage goes wider: TechCrunch reported that the defaults also reach new sites set up by existing customers and existing Free-tier customers, and Help Net Security summarised it similarly. The two accounts differ — if you're on Free, check your own dashboard rather than trusting either.

You can opt out. Cloudflare says that "if a website owner wants to opt out of these new default configurations, they can easily mark this in their Security settings any time leading up to September 15," and that it will keep notifying customers as the date approaches.

So the default itself is narrower than the headlines. The genuinely dangerous change is somewhere else entirely.

The mixed-use trap: one checkbox, two outcomes

After September 15, category rules are "enforced by the most restrictive applicable rules." That sounds like housekeeping. It isn't. Cloudflare spells out the consequence:

"Multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training."

Googlebot is classified as Search and Training, because Google's own crawler documentation describes Googlebot as used "to find information for building Google's search indexes, perform other product specific crawls, and for analysis." One identity, several jobs. When the most restrictive rule wins, a Training block catches the whole identity — and takes Google Search indexing with it.

This is not a hypothetical for later in the year. Reporting on the announcement notes that OpenAI has already split its crawlers by purpose while Googlebot has not, with a fuller migration not expected until later in 2026. The trap is live on day one.

Google hasn't ignored the problem — it built a different escape hatch. Search Engine Land reported in 2026 that Google added a Generative AI performance report in Search Console plus a toggle removing a site from AI Overviews, AI Mode and AI Overviews in Discover, with Google stating that "sites that opt out will not receive traffic or impressions from our generative AI features" while standard rankings are unaffected. It's a per-surface opt-out rather than a split crawler.

The practical takeaway: if you want to refuse training, refuse it bot by bot, not category-wide. Category-level blocking is a blunt instrument while the largest crawlers are still multi-purpose.

Three questions that settle every bot

Strip away the vendor branding and every AI crawler decision comes down to three questions:

1. Does it train? Content goes into a model's weights. No attribution, no link, no traffic.
2. Does it retrieve? The bot builds an index the engine consults at answer time. This is the pipe your citations travel through.
3. Does it act for a user? Someone pasted your URL into ChatGPT or Claude and asked about it. That's a person, routed through a model.

Most published debate treats these as one decision. They aren't, and for most vendors they haven't been for a while: refusing training while keeping retrieval and agent access open is a supported configuration at OpenAI, Anthropic and Perplexity today.

The bot-by-bot reference

Each row below is what the vendor's own documentation says, plus what blocking actually costs you.

Bot	Vendor	Category	What blocking it costs
GPTBot	OpenAI	Training	Excluded from foundation-model training. No documented effect on ChatGPT search appearance.
OAI-SearchBot	OpenAI	Search	"Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers."
ChatGPT-User	OpenAI	Agent	Live fetches when a user asks ChatGPT to visit your page.
ClaudeBot	Anthropic	Training	Future material excluded from Claude training data.
Claude-SearchBot	Anthropic	Search	Reduced visibility and accuracy in Anthropic's search results.
Claude-User	Anthropic	Agent	Your page isn't fetched when a Claude user's question needs it.
Googlebot	Google	Search and Training	Blocking Training after Sept 15 blocks Googlebot — i.e. Google Search.
Google-Extended	Google	Training token	Excluded from Gemini training and grounding. Does not affect Google Search, and does not control AI Overviews.
Applebot / BingBot	Apple / Microsoft	Mixed-use	Named by Cloudflare as blocked when Training is blocked.
PerplexityBot	Perplexity	Search	Blocking removes you from Perplexity results.
Perplexity-User	Perplexity	Agent	User-triggered fetch; generally ignores robots.txt.
CCBot	Common Crawl	Training	Excluded from the open corpus many model builders ingest.
xAI-Bot / GrokBot	xAI	Undocumented	Unresolved — see below.

The load-bearing rows come straight from vendor docs. OpenAI states that disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models," and separately that opting out of OAI-SearchBot removes you from ChatGPT search answers. Perplexity says PerplexityBot is "designed to surface and link websites in search results on Perplexity" and is "not used to crawl content for AI foundation models." Anthropic updated its crawler docs on February 20, 2026 to describe its three bots by purpose, warning that "alternate methods like blocking IP address(es) may not work correctly or persistently guarantee an opt-out."

Two rows deserve extra care.

Google-Extended is narrower than people assume. It governs Gemini training and grounding in Gemini apps. It does not control AI Overviews or AI Mode, which draw on the regular search index via Googlebot. Blocking it to "get out of AI Overviews" doesn't achieve that.

If your goal is to avoid appearing in AI Overviews while preserving regular search rankings, follow the practical steps in how to opt out of AI Overviews without losing rankings.

Grok is the ragged edge. Practitioner write-ups describe xAI documenting user agents in the xAI-Bot / GrokBot family while little real traffic appears under them, and Grok's answers are grounded partly in the X firehose, which reaches xAI through internal integration rather than web crawling. No official xAI crawler documentation was located during this research, so treat that as unresolved. For Grok, your crawler configuration tells you very little and measured answers tell you almost everything.

One asymmetry underpins all of it: robots.txt is a request; a Cloudflare edge block is enforcement. OpenAI notes that because ChatGPT-User actions are user-initiated, "robots.txt rules may not apply," and Perplexity says the same of Perplexity-User. A network-level block returns a 403 and stops them regardless. Cloudflare's own managed robots.txt reference puts it plainly: "robots.txt compliance is voluntary." Flipping the Agent category off does something robots.txt never could — in both directions.

What the numbers say about the trade

The case for blocking is real, and Cloudflare's own network data makes it. Across a 12-month average through July 2025, Cloudflare found AI crawl traffic split 80% training, 18% search, 2% user actions — over the trailing six months, 82/15/3. The same analysis tracked crawl-to-refer ratios across a fixed set of news-related customers: between January and July 2025, Anthropic moved from 286,930:1 to 38,066:1, OpenAI from 1,217:1 to 1,091:1, Perplexity from 54:1 to 195:1, and Google from 3.8:1 to 5.4:1. Cloudflare publishes a caveat worth repeating — traffic referred by Claude's native app doesn't include a Referer header, which inflates Anthropic's ratio. Even discounted, the picture holds: AI crawlers take far more than they send back, and TechCrunch reported that over 50% of AI crawl traffic re-fetches unchanged pages.

The case for staying open is about citations, not clicks. In a July 6, 2026 study of 1,058 prominent domains, cloro found that the median GPTBot-blocking domain earned 0.003 ChatGPT citations per Google ranking it holds, versus 0.417 for non-blocking domains. cloro is careful about what that shows: "the data can't prove causation — domains that block crawlers may differ in other ways." What makes it credible is the engine-specific pattern — blocking OpenAI's crawlers tracks with vanishing ChatGPT citations, blocking Perplexity's with vanishing Perplexity citations.

Where clicks do happen, they behave unusually well. Ahrefs published its own site data in June 2025: AI search was 0.5% of sessions but 12.1% of signups — roughly 23x the conversion rate of organic search — while AI-search users clicked links 75% less often than organic users. Ahrefs flags that the finding may be specific to its industry. Similarweb's generative AI statistics put ChatGPT referral conversion at 7.1%, second only to paid search at 7.8%.

And blocking rates are a scope minefield. Four numbers circulate, all defensible with their scope attached and misleading without it:

• 35.7% of the top 1,000 sites disallowed GPTBot in robots.txt — Originality.ai, August 2024.
• 13.9% of 1,058 high-authority domains disallowed GPTBot — cloro, July 2026. Retrieval bots fared far better: OAI-SearchBot 3.4%, ChatGPT-User 6.2%.
• 4.71% of a broad-web robots.txt sample disallowed GPTBot by May 2026, down from 5.52% in Q1 — technologychecker's analysis.
• 40.9% of the top 1,000 sites were unreadable to GPTBot in a live-fetch test rather than a robots.txt read — a July 2026 Vidern study reported by AuthorityTech. (This one comes via a secondary aggregator; the primary publication wasn't located.)

Blocking is concentrated among large publishers, and it is drifting down among the domains that AI engines actually cite. But the single most useful figure in that list is the one nobody quotes: in the same live-fetch test, 17.6% of top sites allow GPTBot in robots.txt and then return 403 Forbidden when GPTBot actually requests a page. Your published policy and your enforced policy are two different systems, and most sites have never checked whether they agree.

The honest case for blocking

None of this makes blocking irrational. If your business model is ad impressions on pages a human is supposed to land on, the trade Cloudflare's data describes is genuinely bad — and Cloudflare's ad-page trigger is a reasonable proxy for exactly that business. Three further arguments deserve to be taken seriously rather than waved off.

AI referral volume is still small. Ahrefs put AI search at 0.5% of its own sessions. The conversion numbers are striking, but they sit on a small base. The counter is that citation isn't referral — being named in an answer has value even when nobody clicks. If you need traffic this quarter, that's a slow argument.

Declarative signals may be inert. Cloudflare's Content Signals Policy adds a Content-Signal: line to robots.txt with search, ai-input and ai-train values, and the July 2026 update adds a use= parameter taking immediate, reference (the default) or full. But Google's John Mueller said in a July 2026 r/TechSEO thread that the directive has, to his knowledge, no effect on any crawler or LLM, and that Google doesn't use llms.txt. Matthew Prince told The Information that Google was given a heads-up but hasn't said whether it will respect the signals. If you want enforcement, you need the edge, not a declaration.

Centralisation is a legitimate worry. Writing in Tech Policy Press in July 2025, Luke Hogg and Tim Hwang argued that "by making it trivially easy to paywall the entire internet, Cloudflare effectively fragments the web into sealed-off silos", citing Cloudflare's 82%-plus share of the DDoS and bot-protection market. That's a serious argument from people who aren't AI-company partisans.

Meanwhile the ecosystem is building things other than walls. Cloudflare and OpenAI announced a research pilot on July 8, 2026 in which Cloudflare shares real-time network signals — content freshness, traffic quality, actual page changes — to make AI search indexing more accurate. Cloudflare frames the programme as "neutral by design" and is running Pay Per Use experiments with Ceramic.ai and You.com that pay site owners when content is referenced. Large publishers including People Inc., Ziff Davis, Yahoo, BuzzFeed and Vox Media have taken a third route, joining the Really Simple Licensing Collective.

The 30-minute audit before September 15

Do these in order. The order matters, because each step checks a layer that can silently override the one below it.

1. Find out what your edge actually does today (10 minutes). Not what robots.txt says — what happens when a named bot requests a page. Cloudflare's AI Crawl Control (formerly AI Audit) is available on all plans including Free, and the Crawlers tab shows per-crawler request volumes with an Actions column for Allow or Block. If you see a crawler with zero allowed requests and a healthy blocked count, you've found your version of the law firm's problem. Our free AI Crawler Checker runs the same question from the outside across GPTBot, ClaudeBot and eight other AI crawlers — a live fetch, not a robots.txt read, which is the distinction the 17.6% figure exists to make.

2. Separate the training decision from the retrieval decision. Write down two answers, not one: "do we want our content in training corpora?" and "do we want to be citable in AI answers?" A perfectly coherent policy is no to the first and yes to the second. That's roughly what the agency Pixelmojo did in early 2026, blocking a dozen training crawlers with Disallow: / while leaving OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User open — then revising the policy in May 2026 to selectively re-admit training bots feeding engines they wanted citations from. No before/after numbers are published, so read it as a real configuration rather than proof of an outcome.

3. Set the Cloudflare categories deliberately, and check the opt-out. If you're on a new domain, or on Free and unsure whether the wider reported scope applies to you, open Security settings before September 15 and make the choice explicitly. Given the mixed-use trap, prefer per-bot Block actions over a blanket Training category block for as long as Googlebot, Applebot and BingBot remain multi-purpose.

4. Read your robots.txt as a crawler would. Cloudflare's managed robots.txt prepends its directives before your existing file, so your Allow lines can end up sitting below a Cloudflare Disallow for the same agent. Its managed block list names ClaudeBot, GPTBot, Google-Extended, Amazonbot and Bytespider among others. Also check you haven't pattern-matched a family: a rule meant for ClaudeBot that also catches Claude-SearchBot and Claude-User blocks exactly the two bots you meant to keep.

5. Don't let llms.txt stand in for access. Adoption grew 8.8x in twelve months, but 97% of llms.txt files received zero requests in May 2026 across an Ahrefs sample of 137,000 domains, and only 1.1% of requests to those files came from AI retrieval bots. llms.txt is cheap and narrow. Crawler access is the thing that decides whether an engine can read you at all. Geoptimizer's free GEO Site Audit checks crawler access, llms.txt, structured data and content shape in one pass, so you can see which layer is actually the constraint.

If you're going to author an llms.txt, follow this short, practical setup that walks through creating and publishing an effective file in twenty minutes: how to set up llms.txt in 20 minutes.

6. Think about the agent side before you block it. A small developer test on July 10, 2026 across twelve sites found naive agent browser requests hit Cloudflare 403s with a summarisable HTML body 100% of the time, with no signal to the agent that it had failed — so an assistant can confidently summarise your block page instead of your content. It's a self-run test, not a benchmark, but the failure mode is worth knowing. Blocking Agent on an ad-supported news page is defensible; blocking it on a product site where you'd like an assistant to complete a task works against you.

Then measure whether it worked

Here's the uncomfortable part, and it's the same reason a law firm went seven months without noticing: crawler changes are invisible in analytics. There is no line in GA4 labelled "citations you didn't get" — what analytics can capture about AI-driven visits, and how to instrument it, is a separate exercise covered in AI Visibility vs AI Traffic: Connect Your Score to GA4. Access is necessary but not sufficient — letting a bot in doesn't mean you get cited, and the only proof a configuration change worked is whether your brand still shows up when buyers ask.

That's the gap Geoptimizer is built for. It runs your buyers' real questions on ChatGPT, Gemini, Claude and Grok with web search enabled, then reports how often you're mentioned and cited as a single 0–100 AI Visibility Score. The split between mention rate and citation rate maps directly onto the distinction this post is built around: you can keep being mentioned from a model's training knowledge while your citations quietly disappear because a retrieval bot stopped getting through. One number moves, the other doesn't, and only a decomposed score shows which. The formula — 35% mention rate, 25% citation rate, 20% prominence, 20% sentiment — is published and versioned, with a changelog.

For a fuller framework that turns AI Visibility scores into practical site tactics, see what Generative Engine Optimization is and how to optimize for it.

The routine around September 15: take a baseline scan now, make your crawler decisions, then re-scan a few weeks later and compare per-engine. Grok matters most here, because it's the engine where crawler configuration tells you the least — and it's tracked as a first-class engine in every plan, including Free.

FAQ

Does Cloudflare block all AI crawlers by default from September 15, 2026? No. Cloudflare's wording is that for new domains onboarding, Training and Agent will be blocked by default on pages that display ads, while Search remains allowed by default. Press coverage reports the change also reaches existing Free-tier customers, which is broader than Cloudflare's own phrasing — worth checking your dashboard rather than assuming either version.

Will blocking AI training crawlers remove me from AI answers? Not by itself. OpenAI's documentation ties GPTBot to foundation-model training with no stated effect on ChatGPT search appearance, and Google states that Google-Extended doesn't affect inclusion in Google Search or act as a ranking signal. The bots that decide citation visibility are the retrieval bots — OAI-SearchBot, Claude-SearchBot, PerplexityBot — plus Googlebot itself.

Why would blocking Training also block Googlebot? Because Cloudflare enforces the most restrictive applicable rule across all of a crawler's declared behaviours, and Googlebot is classified as both Search and Training. Cloudflare names Googlebot, Applebot and BingBot as crawlers that will be blocked by customers who choose to block Training. Until those crawlers are split by purpose, per-bot rules are safer than a category-wide Training block.

Is robots.txt enough to control AI crawlers? It's a request, not enforcement — Cloudflare's own documentation notes that "robots.txt compliance is voluntary," and both OpenAI and Perplexity say robots.txt may not apply to their user-initiated agent fetches. An edge rule is enforcement. That asymmetry cuts both ways: it's why an edge block is effective, and why an edge block you didn't know about can be silently costing you citations.

Before September 15, check what your edge actually does

The most common failure here isn't choosing wrong. It's never choosing at all — inheriting a default from a migration, a managed robots.txt, or a checkbox someone ticked in 2024, and then reading analytics that will never show the cost.

Take half an hour this week. Run the free AI Crawler Checker and GEO Site Audit to see which of the ten major AI crawlers can actually reach your pages, then take a baseline AI Visibility scan across ChatGPT, Gemini, Claude and Grok so you can tell in September whether your crawler decisions cost you anything. The Free plan covers 3 AI-suggested tracked prompts across all four engines with no credit card and no expiry; pricing starts at $19/month when you need more. Decide deliberately, then verify — because you still can't see traffic that never arrives.