Cloudflare AI Agent Verification — Bot Management, Web Bot Auth, and the Trust Layer Between Your Store and the Open Internet.

Why User-Agent strings and IP ranges are not authentication.

When a request arrives at your Cloudflare-proxied origin with User-Agent: ChatGPT-User, you have a genuine identity problem. The User-Agent header is self-declared and unenforceable: any Python script can write that string in one line. Historically, operators supplemented user-agent inspection with IP address range validation — checking whether the source IP belongs to a CIDR block published by OpenAI, Anthropic, or Google. That mechanism is structurally brittle.

IP ranges shared across a major cloud provider's infrastructure can be claimed simultaneously by multiple tenants. They change without warning as the underlying fleet rotates. Any scraper operating from the same cloud provider — or routing through a VPN or residential proxy network — can appear to originate from the same CIDR blocks as a legitimate crawler. As Cloudflare's engineering team stated directly in their May 2025 Web Bot Auth announcement: "the logic around IP address ranges representing a product or group of users is brittle." That is not a competitive claim — it is an accurate description of how anycast IP space actually works.

robots.txt compounds the enforcement gap. It is a policy document — it declares what bots should do under a voluntary social convention. It cannot prevent any actor from reading it and ignoring it. A scraper that spoofs ChatGPT-User and disregards your robots.txt directives has violated a convention, not bypassed any technical control.

This is the gap that Cloudflare AI agent verification fills. The trust layer operates at the reverse proxy — before traffic touches your origin — and stacks three reinforcing mechanisms: cryptographic HTTP Message Signatures (Web Bot Auth), machine-learning bot scoring, and the Verified Bots / Signed Agents programs. Together, they move identity verification from policy to cryptography.

The spec stack: what is ratified, what is still a draft.

RFC 9421 — The Foundation

RFC 9421 — HTTP Message Signatures — was published by the IETF in February 2024, authored by A. Backman, J. Richer, and M. Sporny. It is an active, ratified specification, not a draft. RFC 9421 defines how to create and verify cryptographic signatures over HTTP message components: request method, target URI, query string, individual headers, and optionally the request body. The signing algorithm, component identifier format, and verification procedure are all normatively defined by the RFC — this is the stable layer that Web Bot Auth builds on top of.

Web Bot Auth — The Application Profile

Web Bot Auth is Cloudflare's application-layer profile of RFC 9421, scoped specifically to bot and agent authentication. The architecture spec — draft-meunier-web-bot-auth-architecture — is an Internet-Draft at the IETF Datatracker, currently at revision -05 as of this writing. It is not endorsed by the IETF as a standard, though Cloudflare has stated the IETF is considering forming a working group around it. The profile adds three things on top of RFC 9421: the tag="web-bot-auth" parameter in Signature-Input, the Signature-Agent header pointing to the bot's key directory domain, and the well-known URL convention at /.well-known/http-message-signatures-directory for public key discovery. (re-verify IETF draft status before launch)

How Request Signing Works

The mechanism follows a public-key infrastructure model with a well-known key directory — no per-site shared secret, no pre-registration with individual websites:

1. Key generation. The bot operator generates an Ed25519 key pair. The JWK-formatted public key is published at /.well-known/http-message-signatures-directory on their domain (e.g., operator.openai.com).
2. Request signing. Before each HTTP request, the bot signs the target authority (@authority) and the signature-agent header using the private key. The Signature-Input header encodes the validity window (created and expires timestamps), the key ID (a JSON Web Key Thumbprint), and the tag="web-bot-auth" parameter.
3. Verification at the edge. Cloudflare's edge checks for the presence of Signature, Signature-Input, and Signature-Agent headers. It fetches the corresponding public key from the bot's key directory, reconstructs the signature base, and verifies the signature using Ed25519. The expires parameter guards against replay attacks — a captured signed request cannot be reused after the expiration timestamp.

GET /path/to/resource HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 Chrome/113.0.0 MyBotCrawler/1.1
Signature-Agent: signer.example.com
Signature-Input: sig=("@authority" "signature-agent");\
               created=1700000000;\
               expires=1700011111;\
               keyid="ba3e64==";\
               tag="web-bot-auth"
Signature: sig=abc==

The Signature-Agent header points to the key directory: crawler.search.google.com for Google Search, operator.openai.com for OpenAI Operator, workers.dev for Cloudflare Workers-hosted agents. The expires timestamp — typically 5 minutes (300 seconds) after creation in reference implementations — is what prevents replay attacks on captured signatures.

Implementation Libraries

Cloudflare publishes two open-source libraries for signing and verifying Web Bot Auth requests:

• web-bot-auth TypeScript npm package — for Node.js bots and Chrome extension-based agents (GitHub: cloudflare/web-bot-auth)
• web-bot-auth Rust crate — for high-performance server-side signing

A live debugging environment is available at https://http-message-signatures-example.research.cloudflare.com for testing implementations against real verification logic.

Integration into the Verified Bots Program

As of July 2025, Cloudflare integrated HTTP Message Signatures directly into its Verified Bots program. Bots applying with well-formed Message Signatures are prioritized and approved more quickly than those using legacy IP validation. Once approved, Cloudflare automatically validates signatures at the edge across all plans. For site owners, no additional configuration is required — if signature validation passes, the traffic is marked verified and available as a WAF rule field. Message Signature verification for origins was available for Free and Pro plans as of this writing, with Business and Enterprise rollout in progress. (re-verify before launch)

Plan tiers, bot score mechanics, and what Enterprise actually unlocks.

Cloudflare's bot protection is not a single product — it is a tiered capability stack. Understanding which tier you are on is prerequisite to understanding which WAF fields and rule expressions are available to you. The most common operator mistake is writing WAF rules that reference cf.bot_management.score on a Pro plan, where those fields simply do not exist.

Bot Score: The ML Engine

The bot score runs from 1 to 99 and is the output of up to four layered detection engines. The scale represents the estimated probability of human origin — lower scores mean more likely automated.

Three detection engines feed into the score: Heuristics — pattern matching against a database of known malicious fingerprints, setting score to 1 for high-confidence automated detections; Machine Learning — the primary engine for sophisticated bots, producing scores in the 2–99 range based on billions of requests processed daily; JavaScript Detections (JSD) — lightweight client-side injection that detects headless browsers. A fourth optional engine, Anomaly Detection, learns a domain-specific baseline and flags outliers. Cloudflare explicitly advises against enabling Anomaly Detection on commerce domains with API traffic or significant traffic variability — the false positive rate in those scenarios is high.

The __cf_bm cookie smooths bot scores across a user session, reducing false positives for real users who exhibit browsing patterns the ML engine might otherwise score below 30. It is set automatically by Cloudflare — no configuration required on your part.

JA3 / JA4 Fingerprinting

cf.bot_management.ja3_hash and cf.bot_management.ja4 are available as WAF rule fields in Bot Management for Enterprise. JA3 is a TLS fingerprint derived from the client hello — unique enough to identify specific bot implementations even when they rotate user agents or IP addresses. A scraper running from a cloud host that spoofs ChatGPT-User will still present a JA3 fingerprint consistent with a scripted HTTP client (Python's requests library, curl, or a headless Chromium build), not with OpenAI's production crawler infrastructure. This fingerprint mismatch is the core detection mechanism in the worked example in §8.

AI Labyrinth

Available on all plans including Free, AI Labyrinth is a next-generation honeypot that serves AI-generated nonsense content to unauthorized scrapers, drawing them into an infinite maze of fake pages. Human visitors and compliant bots never see these pages — AI Labyrinth content is served only to bots that follow unlinked honeypot links. Bots that enter the labyrinth are identified, and their fingerprints are fed back into Cloudflare's ML training pipeline, improving detection for all customers globally. Enable it via Security → Bots → Configure Bot Fight Mode → Enable AI Labyrinth.

From visibility to monetization — the crawl economics layer.

AI Crawl Control (Formerly AI Audit)

AI Crawl Control is available on all Cloudflare plans with zero configuration required. It provides a real-time dashboard showing which AI crawlers are hitting your domain, what URLs they are fetching, whether they are respecting your robots.txt directives, and whether they are classified as buyer agents, search indexers, or training crawlers. The dashboard navigation changed from "AI Audit" to "AI Crawl Control" — re-verify the current navigation path in your dashboard, as Cloudflare updates product naming frequently.

Pay-Per-Crawl (Private Beta)

Pay-Per-Crawl is currently in private beta. Cloudflare acts as Merchant of Record — you set a price per request for your domain, and Cloudflare handles billing aggregation and financial settlement with registered crawler operators. The technical mechanism repurposes HTTP 402 Payment Required.

# 1. Crawler requests resource without payment intent
GET /products/running-shoes.html HTTP/1.1
Host: www.your-store.com

# 2. Cloudflare responds: payment required at stated price
HTTP/1.1 402 Payment Required
crawler-price: USD 0.001

# 3. Crawler retries with exact-price confirmation
GET /products/running-shoes.html HTTP/1.1
Host: www.your-store.com
crawler-exact-price: USD 0.001

# 4. Cloudflare charges and responds with content
HTTP/1.1 200 OK
crawler-charged: USD 0.001
server: cloudflare

# 1. Crawler sends request with max-price header
GET /products/running-shoes.html HTTP/1.1
Host: www.your-store.com
crawler-max-price: USD 0.002

# 2a. If configured price <= crawler-max-price: charge and respond
HTTP/1.1 200 OK
crawler-charged: USD 0.001
server: cloudflare

# 2b. If configured price > crawler-max-price: payment required
HTTP/1.1 402 Payment Required
crawler-price: USD 0.005

Publisher controls allow setting a flat per-request price across the entire domain — path-level pricing was not available in the beta as of this writing (re-verify). For each known crawler, publishers choose Allow (free access), Charge (at the configured price), or Block (no access). If a crawler has no billing relationship with Cloudflare, choosing "Charge" is functionally equivalent to HTTP 403 — the crawler is told a paid relationship is possible, but no content is served.

Pay-Per-Crawl enforcement runs after existing WAF policies and Bot Management rules — your current security posture is not bypassed to charge a crawler. Crawler operators must register with Cloudflare via Web Bot Auth (Ed25519 key pair, public key directory, HTTP Message Signatures on each request) to participate. This registration requirement is the anti-spoofing control — a bad actor cannot impersonate a registered crawler and trigger fraudulent payments because the cryptographic signature cannot be forged without the private key.

Three categories, different economics — and a fourth just arrived.

The single most operationally consequential distinction in Cloudflare AI agent verification is that bots from the same AI provider — even from the same company like OpenAI — serve fundamentally different purposes and should receive different access policies. The taxonomy anchors on three OpenAI bots as the clearest example, then extends to all providers.

Cloudflare's Five Responsible AI Bot Principles

Cloudflare's responsible AI bot principles framework (published September 2025) formalizes this taxonomy into five required behaviors for bots that want to be classified as "good actors." Cloudflare's position — stated directly — is that combining multiple declared purposes in a single bot forces operators into false trade-offs and violates Principle #3 (single declared purpose). The three declared purposes are: Search (building an index, providing results), AI-input (real-time retrieval-augmented grounding for generative answers), and Training (fine-tuning models). (re-verify principle list as Cloudflare publishes updates)

Signed Agents: A Fourth Category (August 2025)

As of August 2025, Cloudflare introduced a Signed Agents classification for user-directed agents that use Web Bot Auth signatures and comply with Cloudflare's signed agent policy. The first cohort includes ChatGPT agent, Goose from Block, Browserbase, and Anchor Browser. Signed agents differ from verified bots in that they are user-directed: the same agent platform (e.g., Browserbase's remote browser infrastructure) may be used by thousands of different end users, each directing the agent toward different goals. The Signed Agents program cryptographically attests to the platform, not the individual user — individual user identity remains an application-layer concern for OAuth (see /agentmall_spoke_oauth). Enterprise customers can take action on signed agents as a group in security rules. (re-verify rule field availability — announced as "coming soon" in August 2025)

Edge-native MCP hosting with bot protection built in.

Cloudflare Workers provides the fastest path to a bot-management-protected MCP endpoint without managing servers. Deploying an MCP server on Workers means your server runs inside Cloudflare's global network — co-located with the same bot detection pipeline that evaluates every incoming request. The WAF rules and Bot Management policies you have configured on your zone apply to agent traffic hitting your MCP endpoint before any application code runs.

Workers: The Bot-Protected MCP Endpoint

A Workers-hosted MCP server at https://your-store-mcp.workers.dev/sse is immediately behind Cloudflare's WAF. The workers-oauth-provider library handles the OAuth 2.1 provider side — Dynamic Client Registration (RFC 7591) and Authorization Server Metadata (RFC 8414) — so your MCP server can authenticate agent clients out of the box. Upstream OAuth access tokens (e.g., from Shopify or GitHub) are stored encrypted in Workers KV; only a narrower, scoped token is issued to the MCP client, limiting blast radius if that token is compromised. In April 2025, Cloudflare announced generally available remote MCP servers, with partnerships with Auth0, Stytch, and WorkOS for authentication and authorization. (re-verify GA status and partner list)

# Free tier
Requests:    100,000 / day
CPU time:    10 ms per request
Memory:      128 MB

# Paid Workers plan (Workers Paid)
Requests:    No cap (billed per million after included)
CPU time:    30 seconds default; up to 5 minutes for long-running
Memory:      128 MB standard; higher limits available

Tunnel: Private Origin with Public MCP

If your commerce logic lives on a private origin — an on-premise server, a private VPC, or Shopify's internal API behind a network boundary — Cloudflare Tunnel lets you expose a public MCP endpoint without a publicly routable IP on your origin. The cloudflared daemon runs on your private infrastructure and creates outbound-only connections to Cloudflare's global network. Your firewall only needs to allow outbound traffic from cloudflared; all inbound connections are blocked at the network level. Publicly, the MCP endpoint resolves to Cloudflare, which applies your full WAF and Bot Management ruleset before proxying traffic through the tunnel.

AI Agent
   ↓
Cloudflare Edge
  ├─ Web Bot Auth signature verification (RFC 9421)
  ├─ Bot score: ML + Heuristics + JSD
  ├─ WAF custom rules (Bot Management Enterprise)
  └─ AI Crawl Control category classification
   ↓
[Cloudflare Tunnel (cloudflared)]
   ↓
Private Origin
  ├─ Shopify Storefront API
  ├─ WooCommerce REST API
  └─ Custom headless commerce backend
   ↓
OAuth 2.1 app-layer identity
(see /agentmall_spoke_oauth)
   ↓
/agents.json capability manifest
(see /agentmall_spoke_agents_page)

Zero Trust MCP Server Portals (Open Beta, August 2025)

For enterprise deployments, MCP Server Portals — part of Cloudflare One / SASE, announced August 2025, open beta — provide a centralized gateway for all MCP connections. This layer requires corporate identity provider authentication (Cloudflare Access) before an agent reaches any MCP server, enforces MFA and device posture checks, provides aggregated MCP request logs across all servers, and presents each user with only the curated list of servers and tools they are authorized to use. This is the enterprise analog of OAuth scope enforcement, but operating at the network layer before HTTP. (re-verify open beta status and feature set)

How bot identity, OAuth, and capability discovery form a complete stack.

The edge bot verification layer is not the complete picture — it is the first gate in a four-layer stack. Understanding where Cloudflare's role ends and where OAuth (application-layer identity) begins is critical for building a coherent trust architecture. Cloudflare verifies who is at the edge. OAuth verifies what that actor is authorized to do once they pass the edge.

Request Flow Diagram

Internet Request
       ↓
[Cloudflare Edge]
  ├─ Web Bot Auth: Does request carry valid RFC 9421 signature?
  │    YES → Mark cf.bot_management.verified_bot = true (or signed_agent)
  │    NO  → Pass to ML + Heuristics pipeline
  ├─ Bot Score: 1–99
  │    1 = definitely automated → Block (if rule configured)
  │    2–29 = likely automated  → Challenge or Block
  │    30–99 = likely human     → Allow
  ├─ AI Crawl Control: Which category is this?
  │    AI Assistant (Buyer)   → Allow / no challenge
  │    AI Search (Indexer)    → Allow or Pay-Per-Crawl
  │    AI Training (Crawler)  → Block or Pay-Per-Crawl
  └─ WAF Custom Rules: Path-specific overrides
       /api/products  → allow cf.verified_bot_category = "AI Assistant"
       /api/checkout  → require OAuth token (→ /agentmall_spoke_oauth)
       /sitemap.xml   → allow all verified bots
       ↓
[Origin / MCP Server / Workers]
  ├─ OAuth 2.1 identity (app-layer): what can this agent do?
  │    (see /agentmall_spoke_oauth)
  └─ Response + /agents.json capability manifest
       (see /agentmall_spoke_agents_page)

Two requests, same User-Agent, different outcomes.

Request A — Legitimate ChatGPT-User (OpenAI ChatGPT Agent)

1. OpenAI's ChatGPT agent sends GET /products/running-shoes with User-Agent: ChatGPT-User, Signature-Agent: operator.openai.com, and a valid Ed25519 Signature and Signature-Input header.
2. Cloudflare's edge detects the Web Bot Auth headers and begins signature verification.
3. Cloudflare fetches the key directory at operator.openai.com/.well-known/http-message-signatures-directory, retrieves the public key matching the keyid parameter, and verifies the signature using Ed25519.
4. Signature verification passes. The request is marked cf.bot_management.verified_bot = true with cf.verified_bot_category = "AI Assistant".
5. WAF Rule 1 (cf.bot_management.verified_bot → Skip all remaining custom rules) fires. Request passes through to origin.
6. Origin serves product page. ChatGPT agent delivers product information to the user who initiated the query.

Request B — Scraper Spoofing ChatGPT-User

1. A Python requests script sends GET /products/running-shoes with User-Agent: ChatGPT-User. No Signature, Signature-Input, or Signature-Agent headers are present.
2. Cloudflare's edge checks for Web Bot Auth headers — absent. Falls through to ML + Heuristics pipeline.
3. ML engine analyzes request features: no browser-consistent TLS fingerprint (JA3 hash matches a known Python/requests pattern, not a browser or OpenAI's known JA3 fingerprint); no browser session signals; cold-start request pattern. Bot score: 2.
4. WAF Rule 3 fires: (cf.bot_management.score gt 1 and cf.bot_management.score lt 30) → Managed Challenge.
5. The script cannot pass a Managed Challenge — it requires JavaScript execution in a real browser context. Request is abandoned.
6. If AI Labyrinth is enabled and the scraper previously followed any honeypot links, its fingerprint may already be in the blocklist — Rule 2 (cf.bot_management.score eq 1 → Block) may fire even before Rule 3.

WAF Rule Configuration (Bot Management for Enterprise)

These are the three baseline rules from Cloudflare's official challenge-bad-bots documentation — real WAF expressions ready for deployment. Rules must be deployed in the order shown; rule order is evaluated top-to-bottom and early exit is critical.

Expression: (cf.bot_management.verified_bot)
Action:     Skip — All remaining custom rules

# This rule must be FIRST. A verified bot (signed agent, registered crawler)
# should bypass all bot score rules. If this rule is placed after score-based
# block rules, verified bots will be incorrectly blocked before skip fires.

Expression: (cf.bot_management.score eq 1)
Action:     Block

# Score 1 = Heuristics engine high-confidence detection.
# Blocking carries very low false-positive risk.

Expression: (cf.bot_management.score gt 1 and cf.bot_management.score lt 30)
Action:     Managed Challenge

# Score 2-29 = ML engine output, likely automated.
# Managed Challenge allows real users through; stops most bots.

Expression: (cf.verified_bot_category eq "AI Assistant" and
             starts_with(http.request.uri.path, "/api/checkout"))
Action:     Skip — All remaining custom rules

# Ensures buyer agents (ChatGPT-User, Claude-User) can reach
# checkout endpoints even on paths with additional restrictions.

Expression: (http.user_agent contains "GPTBot" and not cf.bot_management.verified_bot)
Action:     Block

# Blocks unverified GPTBot; allows verified GPTBot (if you choose to allow it).
# Use this pattern if you want to block training crawlers that are not
# properly registered in Cloudflare's Verified Bots program.

Eight ways Cloudflare bot rules break in production.

1. Wrong rule order — verified bot skip placed after score-based block

The problem: cf.bot_management.score lt 30 catches all likely-automated traffic. AI buyer agents (ChatGPT-User) that are verified bots have cf.bot_management.verified_bot = true — but only if your skip rule fires first. If Rule 1 (skip verified bots) is placed after Rule 2 (block score < 30), verified bots get blocked before the skip executes. Cloudflare's rules engine evaluates in strict top-to-bottom order. The fix: Rule 1 must always be the first rule and must skip all remaining custom rules when cf.bot_management.verified_bot is true. See the baseline in §8.

2. Conflating User-Agent string matching with bot identity verification

The problem: writing a WAF rule like http.user_agent contains "ChatGPT-User" → Allow, and treating that as authentication. Any scraper can write that string. User-agent-based allow rules create a spoofing bypass by design. The fix: use cf.bot_management.verified_bot (Enterprise) or cf.verified_bot_category eq "AI Assistant" for identity-based rules. On non-Enterprise plans, rely on the Block AI bots toggle — maintained by Cloudflare against a curated directory — rather than home-grown user-agent matching.

3. Enabling Bot Fight Mode on an API-first or headless storefront without exempting own traffic

The problem: Bot Fight Mode and Super Bot Fight Mode apply across all traffic on the domain. Automated traffic from your own mobile app, partner APIs, webhook integrations, or health-check monitors will be challenged or blocked. A midnight deployment pipeline hitting your staging endpoint will trigger Managed Challenge at 2am. The fix: on Bot Management for Enterprise, use not starts_with(http.request.uri.path, "/api") exclusions in custom rules. On lower plans, review the bot settings carefully and consider exempting specific IP ranges or JA3 fingerprints for known-good automation before enabling Bot Fight Mode site-wide.

4. Deploying Pay-Per-Crawl without verifying crawler registration

The problem: Pay-Per-Crawl requires both parties — publisher and crawler — to have Cloudflare accounts with configured payment details. A publisher who sets a price for an unregistered crawler will see a functional block (HTTP 403-equivalent behavior), not a paid crawl. Setting "Charge" for an unregistered crawler is ambiguous and may confuse your analytics. The fix: confirm which crawlers have active billing relationships with Cloudflare before configuring "Charge" policies. Use "Block" explicitly for crawlers without relationships. (re-verify registered crawler list as the beta expands — the list is expected to grow significantly)

5. Relying solely on AI Crawl Control and missing unknown scrapers

The problem: AI Crawl Control shows you known, classified AI bots from Cloudflare's directory. It does not show unknown scrapers that are not in that directory. Those appear in Security Analytics and Bot Analytics (Enterprise), not in AI Crawl Control. Operators who use AI Crawl Control as their sole monitoring tool systematically undercount unauthorized scraping. The fix: cross-reference AI Crawl Control data with Security Analytics. Enable AI Labyrinth — it surfaces unknown scrapers by luring them into honeypot paths and adding their fingerprints to the blocklist.

6. Not registering your own bot or agent with the Verified Bots program

The problem: if you operate your own AI agent — a commerce assistant, price comparison bot, or inventory checker — and it visits other Cloudflare-protected sites, it will score poorly on the ML engine and may be challenged or blocked. The ML engine sees an automated HTTP client with no browser signals and assigns a low bot score. The fix: submit your bot via the Verified Bots submission form in your Cloudflare account (Account Home → three dots → Configurations → Verified Bots). If your bot signs requests using Web Bot Auth, select "Request Signature" as the verification method — it is prioritized and approved more quickly than legacy IP validation.

7. Enabling Anomaly Detection on a commerce domain with API traffic or traffic spikes

The problem: Anomaly Detection learns a traffic baseline for your domain and flags outliers as score 1 (definitively automated). On an e-commerce site with large API traffic, promotional traffic spikes, or a Cloudflare for SaaS multi-tenant setup, the baseline is inherently noisy — and real users during a flash sale or product launch may be blocked en masse. Cloudflare explicitly advises against enabling Anomaly Detection on these domain types. The fix: do not enable Anomaly Detection on commerce domains with variable traffic. Use the ML engine and Heuristics pipeline instead — they handle legitimate variability without learning a fixed baseline.

8. Skipping the Signed Agents program consideration for MCP server deployments

The problem: if your MCP server is designed to be called by user-directed AI agents (ChatGPT agent, Claude agent, Goose), and those agents are registered in Cloudflare's Signed Agents directory, you need WAF rules that correctly identify and handle them. If you have generic score lt 30 → challenge rules without a prior signed-agent pass-through rule, those agents will be challenged and your MCP tool calls will silently fail at the network layer — with no error surfaced to the end user. The fix: add an explicit rule to pass signed agents (AI Assistant category + verified_bot) before your generic bot score rules. Monitor the signed agents WAF field when Cloudflare makes it generally available — announced as "coming soon" in August 2025. (re-verify field name and plan availability before launch)

Frequently asked questions.

The 30-day rollout, in five steps.

Each step mirrors the HowTo JSON-LD at the top of this page word for word.

Step 1 — Audit your current AI bot traffic

Before writing any rules, establish a baseline. In the Cloudflare dashboard, navigate to the AI Crawl Control tab for your domain. Review the breakdown by AI provider, bot category, and most-fetched URL paths. Export the CSV. Identify: (a) which AI Assistants (buyer agents) are hitting your product and checkout paths, (b) which AI Search crawlers are indexing your content, (c) which Training crawlers are accessing your domain, and (d) whether any crawlers are ignoring your robots.txt directives. This baseline prevents you from blocking legitimate buyer agent traffic when you tighten rules.

Step 2 — Configure Bot Fight Mode and AI Labyrinth (All Plans)

In the Cloudflare dashboard, go to Security → Bots → Configure Bot Fight Mode. Enable the Block AI bots toggle to block known training crawlers that you do not wish to allow. Enable AI Labyrinth to trap unauthorized scrapers and feed fingerprint data to Cloudflare's ML training pipeline. Enable Instruct AI bot traffic with robots.txt to have Cloudflare enforce your existing robots.txt directives at the edge for bots that are known to Cloudflare's directory. These three toggles require no rule authoring and update automatically as Cloudflare adds new bot signatures.

Step 3 — Deploy baseline WAF rules for bot traffic (Enterprise: Bot Management Add-on)

If you have Bot Management for Enterprise, deploy the three-rule baseline from Section 8 in order: (1) Skip verified bots, (2) Block score = 1, (3) Managed Challenge score 2–29 and not verified bot. Then add path-specific overrides: allow cf.verified_bot_category = "AI Assistant" through to your API endpoints and MCP server path; block or challenge generic automation on /admin, /cart, and /checkout paths. Use Bot Analytics to review the impact on traffic before tightening thresholds.

Step 4 — Deploy MCP server on Cloudflare Workers with OAuth (All Plans)

Create a new Worker using npm create cloudflare@latest -- --template=remote-mcp-authless or the OAuth-enabled template. Add the workers-oauth-provider library for OAuth 2.1 support. Configure your tool definitions to check cf.verified_bot_category or session-level identity claims (from OAuth) before returning sensitive data (inventory, pricing APIs, checkout). Deploy with wrangler deploy. Your MCP endpoint at your-worker.workers.dev/sse is immediately behind Cloudflare's WAF and receives the bot detection fields on every request. For private origins, deploy cloudflared on your infrastructure and connect via a Tunnel, pointing the public-facing Cloudflare hostname to the tunnel.

Step 5 — Register your bot with Cloudflare and implement Web Bot Auth (If You Operate a Bot or Agent)

Generate an Ed25519 key pair. Publish the JWK public key at https://yourdomain.com/.well-known/http-message-signatures-directory. Sign the key directory URL itself using web-bot-auth (TypeScript or Rust library) to prove ownership. Register via the Verified Bots submission form in your Cloudflare account (Account Home → three dots → Configurations → Verified Bots), selecting "Request Signature" as the verification method. Configure every outbound HTTP request from your bot to include Signature, Signature-Input, and Signature-Agent headers. Test your implementation against https://http-message-signatures-example.research.cloudflare.com. Once approved, your bot will pass Cloudflare's signature verification at all sites it visits and will be excluded from challenges.

The trust layer continues at every spoke.

Edge bot verification is the first gate. These spokes cover what happens downstream — identity after the edge, capability discovery, and the full trust framework for agent-ready commerce.