Is Cloudflare Blocking Google or AI Crawlers?

Direct answer: if your site uses Cloudflare, open the live AI bot settings before choosing a blanket block. The current controls distinguish allowing crawlers, asking accountable mixed-use crawlers not to train, blocking AI access on pages with ads, and broader blocking. Use Disallow AI Training when you want search discovery but not model training. Cloudflare says the ordinary Block and Block on pages with ads settings now apply to mixed-use crawlers, including Googlebot, Bingbot, and Applebot, so those choices can also remove search access.

This guide helps a public website avoid an accidental search-visibility loss. It does not protect private data or guarantee that every crawler will respect a preference.

The practical goal is not “block AI.” Keep the discovery channels that can bring customers, then restrict the uses that do not fit the business.

Not sure whether Cloudflare is blocking useful crawlers?

We can check the public crawl path and separate a real access problem from settings that do not need work.

See the fixed-price SEO audit

Choose the setting by the outcome you want

Setting or outcome	Best fit	Main risk
Allow	You want broad search, agent, and AI discovery	Content may be available for uses you would rather restrict
Disallow AI Training	You want search access while expressing a no-training preference	It relies on accountable crawlers honoring the published preference
Block on pages with ads	An ad-supported site wants Cloudflare's bounded publisher option	Mixed-use crawler behavior still needs checking
Block	You intentionally accept the visibility tradeoff	A mixed-use crawler needed for search can be stopped at the edge

September 15 warning: Block is no longer a training-only choice for mixed-use crawlers. Cloudflare says it can stop Googlebot, Bingbot, and Applebot at the edge. Choose Disallow AI Training when the intended outcome is “keep search, refuse training.”

The exact controls shown can depend on the zone and Cloudflare product state. Confirm the dashboard, generated robots.txt, and custom rules instead of copying a screenshot from another account.

Googlebot exposes the policy problem

Cloudflare explicitly named Googlebot as a multi-purpose Search and Training crawler in its July announcement. Google documents the controls differently. Google says Googlebot is used for Search crawling, while Google-Extended is a standalone robots.txt product token controlling whether crawled content may be used for future Gemini training and certain grounding uses. Google also states that Google-Extended does not affect inclusion or ranking in Google Search.

Google-Extended is not a separate HTTP user agent. Requests still arrive through existing Google crawler identities. That explains why an edge platform classifying network traffic and a vendor documenting a product-level robots token can describe the same ecosystem differently.

Decision boundary: blocking Google-Extended in robots.txt is not the same action as blocking verified Googlebot traffic at Cloudflare's edge. The first expresses a product-use preference documented by Google. The second can prevent Google from fetching the page for Search.

Choose the policy by business model

Business goal	Search	Agent	Training	What to validate
Service business seeking discovery	Allow	Allow or monitor	Owner decision	Important service pages remain reachable by Googlebot, Bingbot, and named AI search crawlers
Ad-supported publisher	Allow	Business-model decision	Disallow or block	Mixed-use Search crawlers are not unintentionally edge-blocked
Ecommerce catalogue seeking assistant referrals	Allow	Often allow with safeguards	Owner decision	Product pages, availability data, and checkout boundaries behave as intended
Private customer portal	Not public	Not public	Not public	Authentication and authorization, not crawler policy, protect the content

No row is a universal recommendation. A publisher funded by page views and a merchant seeking product referrals have different incentives. The important step is to write the business outcome before touching the controls.

Configure without breaking search

Record the existing state. Capture the Search, Agent, and Training settings, the legacy Block AI Bots state, Bot Preference Sync, and relevant custom rules.
Choose by use, not by the word AI. Decide whether each public content type should support discovery, real-time user retrieval, and training.
Inspect mixed-purpose consequences. If Training is set to Block, determine whether Cloudflare will also stop a crawler needed for Search.
Review generated robots.txt. Bot Preference Sync prepends rules; existing directives remain, so read the combined file for conflicts and duplicated groups.
Review custom rules separately. A hand-written WAF block or challenge can still deny a crawler that the category policy allows.
Test representative URLs. Check an article, commercial page, static asset, and the live robots.txt.
Monitor real logs after the change. Look for verified crawler responses, 401/403/429 spikes, challenge pages, and crawl declines.

A five-minute review checklist

Write down whether you want search discovery, AI-answer discovery, agents, and training.
Open the current zone settings rather than relying on a default described elsewhere.
Inspect the generated robots.txt and any custom WAF rules.
Confirm that Googlebot and Bingbot are not being treated as training-only traffic.
Check a real important page and review verified-bot analytics after the change.

If you are not using Cloudflare as the active proxy for the domain, these dashboard defaults do not control your traffic. The underlying policy exercise still applies, but enforcement belongs to the CDN, WAF, reverse proxy, or origin actually serving the site.

Sources checked

Product behavior and crawler purposes were verified from official documentation on September 24, 2026. Cloudflare's settings and classifications can change; check the live zone and BotBase before implementing a rule.

Need to separate crawler policy from SEO risk?

FloxoLab can review the public crawl path, edge rules, and measurement setup, then turn the findings into a small, testable change list.

Explore the SEO audit