Cloudflare Syncs robots.txt With Edge AI Crawler Policies

Cloudflare launched Bot Preference Sync on August 21, 2026, connecting a site’s public robots.txt directives with AI crawler policies enforced at Cloudflare’s edge. The feature is available across Free, Pro, Business, and Enterprise plans, giving site operators one dashboard control for crawler preferences.

One Control Plane for Crawler Policy

Bot Preference Sync addresses a common operational gap: robots.txt is advisory, while firewall and bot-management rules determine whether requests reach a site. Separate configurations can drift, leaving a crawler-facing file that says one thing and edge enforcement that does another.

Cloudflare generates policy rules from the zone-level dashboard and serves them through the edge. For an existing robots.txt file, the generated directives are prepended between # BEGIN Cloudflare Bot Preference Sync and # END Cloudflare Bot Preference Sync markers. Existing site rules and Disallow directives remain below the managed block and are not modified.

The generated user-agent targets are refreshed continuously through BotBase, Cloudflare’s internal directory of verified crawlers. This gives the policy a maintenance path as crawler identities change, instead of requiring developers to update static files manually.

Search, Agent, and Training Policies

Cloudflare separates AI crawler activity into three functional categories:

Crawler purpose	Available policies
Search	Allow, Block on pages that serve ads, or Block everywhere
Agent	Allow, Block on pages that serve ads, or Block everywhere
Training	Allow or Disallow

The distinction matters for sites that want visibility in conventional search while limiting model-training use. Agent crawlers retrieve content for task-based AI assistants, which creates a different access pattern from search indexing. Training crawlers harvest data for building or fine-tuning machine-learning models.

If you operate a content site, the dashboard settings should match the business purpose of each access class. Developers maintaining custom crawler rules can also disable Bot Preference Sync and continue managing static files and security logic independently.

Conditions for Mixed-Use Crawlers

A single user-agent may perform both search indexing and model training. When a site selects Disallow Training, such a crawler can retain search access only when it satisfies four transparency conditions.

First, it must directly honor a no-training preference in robots.txt through any mechanism. It must also provide an explicit AI-summary opt-out, offer URL-level analytics showing which pages were available for training alongside search metrics, and publicly demonstrate that refusing training does not reduce traditional search ranking or results.

Cloudflare tracks crawlers that meet these requirements in the AI bot transparency directory on Cloudflare Radar. Mixed-use crawlers that fail the conditions are blocked at the edge when training is disallowed. The approach puts crawler identity and disclosure alongside enforcement, a useful complement to broader work on AI agent website access.

Rollout and Migration Details

New Cloudflare accounts have Bot Preference Sync enabled by default. Existing users of Cloudflare’s legacy managed robots.txt feature will see a dashboard prompt to review and confirm preferences before moving to the synchronized system.

The feature is especially relevant for teams exposing content or tools to agents through infrastructure such as Cloudflare WebMCP. Search, agent retrieval, and training access can now be expressed in one policy layer while edge controls enforce the result.

Review the three crawler categories before accepting the default configuration, test the served robots.txt output, and audit any existing firewall rules for conflicts. Keep synchronization disabled only when a partner-specific arrangement or complex security policy requires separate manual control.