Cloudflare's September 15 AI Bot Rules: Keep Search Open Without Allowing Everything

Direct answer: if your domain uses Cloudflare, review its AI bot policy instead of assuming that “block training” is isolated from ordinary search. Cloudflare's July guidance said that, from September 15, 2026, new domains using the ad-supported onboarding/default path would allow Search while restricting Training and Agent traffic on pages with ads—and that the most restrictive rule could block multi-purpose crawlers such as Googlebot and Bingbot. A newer August update introduced Bot Preference Sync and a less blunt Disallow Training path for cooperating mixed-use crawlers. The two official descriptions do not yet read identically, so the safe action is to confirm the live settings shown for your zone.

This guide is for site owners deciding how public content may be discovered and reused. It does not cover private-data protection, anti-scraping guarantees, or Cloudflare's separate AI Search product.

What changes on September 15

Cloudflare's current bot documentation defines three broad AI behaviors. Search crawlers collect or index content to answer questions later. Agent traffic acts in real time on a person's behalf. Training crawlers collect content for model training or fine-tuning, including crawlers Cloudflare classifies as serving both Search and Training.

The July 1 announcement set the following September 15 defaults for new domains onboarding to Cloudflare: Search allowed, with Training and Agent blocked on pages displaying ads. It also said that customers choosing to block Training could block multi-purpose crawlers under the most restrictive applicable rule.

Cloudflare's guidance evolved before the deadline

The change is unusually easy to misreport because Cloudflare published a refinement after its initial announcement.

That is not proof that Cloudflare will behave unpredictably. It means the product language was still converging close to the effective date. Treat the current dashboard and generated policy for your specific zone as implementation evidence, and retain the dated documentation as the explanation of how you reached the decision.

Googlebot exposes the policy problem

Cloudflare explicitly named Googlebot as a multi-purpose Search and Training crawler in its July announcement. Google documents the controls differently. Google says Googlebot is used for Search crawling, while Google-Extended is a standalone robots.txt product token controlling whether crawled content may be used for future Gemini training and certain grounding uses. Google also states that Google-Extended does not affect inclusion or ranking in Google Search.

Google-Extended is not a separate HTTP user agent. Requests still arrive through existing Google crawler identities. That explains why an edge platform classifying network traffic and a vendor documenting a product-level robots token can describe the same ecosystem differently.

Crawler control matrix

Cloudflare classifications can change as operators disclose more information and Cloudflare updates BotBase. The table therefore separates each vendor's stated purpose from Cloudflare's published statements instead of pretending that every row has one permanent category.

Applebot was also named by Cloudflare as multi-purpose. If Apple search or assistant visibility matters to the business, inspect its current BotBase record and Apple's current crawler documentation before applying a category-wide rule.

Choose the policy by business model

No row is a universal recommendation. A publisher funded by page views and a merchant seeking product referrals have different incentives. The important step is to write the business outcome before touching the controls.

Configure without breaking search

1. Record the existing state. Capture the Search, Agent, and Training settings, the legacy Block AI Bots state, Bot Preference Sync, and relevant custom rules.
2. Choose by use, not by the word AI. Decide whether each public content type should support discovery, real-time user retrieval, and training.
3. Inspect mixed-purpose consequences. If Training is set to Block, determine whether Cloudflare will also stop a crawler needed for Search.
4. Review generated robots.txt. Bot Preference Sync prepends rules; existing directives remain, so read the combined file for conflicts and duplicated groups.
5. Review custom rules separately. A hand-written WAF block or challenge can still deny a crawler that the category policy allows.
6. Test representative URLs. Check an article, commercial page, static asset, and the live robots.txt.
7. Monitor real logs after the change. Look for verified crawler responses, 401/403/429 spikes, challenge pages, and crawl declines.

Content Signals add a second policy layer

Cloudflare's Content Signals vocabulary lets a site express three purposes independently: search, ai-input, and ai-train. The newer experimental use value expresses how much content may be retained or reproduced: immediate, reference, or full.

User-agent: *
Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
Allow: /

This example expresses a preference to allow conventional search and real-time AI input, reserve model training, and permit indexing, excerpts, and links rather than full reproduction. It is not a universal standard enforced by every crawler. Cloudflare says Content Signals express a site owner's preference; technical blocks and cooperating crawler behavior remain separate.

There is now one concrete implementation beyond a declaration: as of August 31, Cloudflare's own Browser Rendering /crawl endpoint checks the purpose and use values and can reject a request that asks for more permission than the target site declares.

Verify the result without faking a bot

A command such as curl -A "Googlebot" is useful for detecting obvious user-agent rules, but it does not prove how Cloudflare treats a verified crawler. User-agent strings are easy to spoof. Cloudflare, Bing, and Perplexity all describe identity or verification methods beyond the header alone.

curl -sS https://example.com/robots.txt
curl -I https://example.com/important-page
curl -I -A "OAI-SearchBot" https://example.com/important-page

Use the first request to inspect the deployed policy, the second to establish the ordinary public response, and the third only as a basic difference check. For stronger evidence, review Cloudflare's verified-bot analytics or logs and use vendor-published IP verification where available. Confirm that the response contains the intended content, not merely a 200 status from a challenge or fallback page.

September 15 review checklist

• The zone is identified as new, existing, ad-supported, or non-publisher.
• Search, Agent, and Training each have an explicit business decision.
• The legacy Block AI Bots setting has been reviewed.
• Bot Preference Sync and the resulting robots.txt have been inspected.
• Googlebot and Bingbot are not assumed to be training-only crawlers.
• Vendor-specific search and training tokens are kept separate where documented.
• Custom WAF rules and challenges agree with the category policy.
• Allowed pages return useful HTML to legitimate crawlers.
• Private pages use real authentication and authorization.
• The configuration has an owner and a dated review record.

If you are not using Cloudflare as the active proxy for the domain, these dashboard defaults do not control your traffic. The underlying policy exercise still applies, but enforcement belongs to the CDN, WAF, reverse proxy, or origin actually serving the site.

Sources checked

Product behavior and crawler purposes were verified from official documentation on September 14, 2026. Cloudflare's settings and classifications can change; check the live zone and BotBase before implementing a rule.

• Cloudflare: Configure AI bot policies
• Cloudflare: Search, Agent, Training, and September 15 defaults
• Cloudflare: Bot Preference Sync and Disallow Training
• Cloudflare: Content Signals Policy
• Cloudflare: /crawl support for content use
• Google: Googlebot and Google-Extended
• OpenAI: OAI-SearchBot and GPTBot controls
• Anthropic: Claude crawler purposes
• Perplexity: search and user-requested crawlers
• Bing: crawler roles and verification