robots.txt and agents.txt for Commerce — Allowlisting the AI Agent Bots That Buy.

Why agents can't find products — and why robots.txt is the first gate.

The AgentMall Roadmap frames the agent-commerce problem as three sequential failures. Failure #1: agents can't find products. Failure #2: products aren't machine-readable. Failure #3: there's no agent-native checkout. The 4-Layer Agent-Ready Model (Structured Data → API → MCP → UCP) addresses Failures #2 and #3 — but only after an agent has found the page. Discovery is the precondition for everything else.

Traditional SEO assumes a specific journey: human → search engine → ranked link → browser click. robots.txt in that model is a contract between you and Googlebot. AI agents break that assumption at two points. First, agents act — they don't merely retrieve. An AI agent completing a shopping task may parse prices, compare attributes, add to cart, or pass structured data to another agent downstream. Whether your product data is findable by that agent is prior to all of that. Second, discovery now happens through multiple pipes simultaneously. A product page may be surfaced to a human by Google Search, to a ChatGPT user by OAI-SearchBot's index, to a Perplexity user by PerplexityBot's index, and retrieved live by Claude-User during an active shopping session — all in the same hour. These pipes use different user-agents, obey robots.txt independently, and can be controlled separately.

Three categories. Three policies. Never a blanket block.

The single most expensive mistake in AI commerce robots.txt configuration is treating all AI bots as a single category. They are not. The three-bucket taxonomy below maps to fundamentally different policy defaults. Allowlisting all AI bots (naive) and blocking all AI bots (equally naive) are both wrong for a commerce site.

Category 1: AGENT BUYERS — Allow unconditionally

These bots fetch content on a real user's behalf during an active task. Not used for automated crawls or model training. The user is waiting. Blocking these bots means your page never enters that user's decision loop — the equivalent of a physical store locking its door the moment a customer arrives.

Category 2: AGENT INDEXERS — Allow broadly

These bots crawl the web to build indexes that AI answer surfaces draw from. Not used for model training. Functionally analogous to Googlebot but for AI answer layers — blocking them has the same effect as blocking Googlebot: you disappear from the results.

Category 3: AI TRAINING CRAWLERS — Operator's call

These bots crawl content to train generative models. Not used for search indexing or user-task retrieval. Blocking them does NOT remove you from AI search — they use different user-agent tokens than the indexers and buyers and serve a completely separate function. See §7 for the full SEO-vs-training tension analysis before deciding.

18 bots, verified against vendor docs. User-agent strings included.

The table below covers all 18 AI bots relevant to commerce operators as of this writing. User-agent string version numbers change without notice — always re-verify against vendor documentation before deploying. The user-agent token (the short keyword) is the robots.txt matching key; the full string appears in HTTP headers and server logs. (Re-verify all user-agent strings before launch.)

Full user-agent strings (for log matching)

User-agent tokens are the keys for robots.txt rules. Full strings appear in server logs. Match by substring — version numbers change. (Re-verify all strings before launch.)

RFC 9309 rules that matter for commerce configuration.

robots.txt was formalized as IETF RFC 9309 in September 2022, codifying rules that most major bots already follow. The rules below have direct commerce implications — getting any of them wrong can silently open paths you meant to close or close paths you meant to open.

A full, deployable file — copy, adapt paths, re-verify versions.

The file below is a complete, runnable robots.txt for an agent-friendly commerce site. Every user-agent token in the file is documented in §3. Adapt the Sitemap URLs and any path-specific rules to match your actual store structure. Re-verify version numbers in user-agent strings before launch — vendors update them without announcement.

# ================================================================
# robots.txt — Agent-Friendly Commerce Configuration
# Last structural review: see Source metadata in your CMS
# Cross-references:
#   /llms.txt       — AI-friendly content layout (see Agent Sitemap spoke)
#   /agents.json    — Agent capability manifest (see /agents page spoke)
# ================================================================

# ----------------------------------------------------------------
# CATEGORY 1: AGENT BUYERS
# These bots fetch pages on a live user's behalf.
# Block them = you disappear from AI-assisted shopping sessions.
# ----------------------------------------------------------------

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Amzn-User
Allow: /

# ----------------------------------------------------------------
# CATEGORY 2: AGENT INDEXERS
# These bots build the indexes that AI surfaces draw from.
# Block them = you disappear from AI answer layers.
# Google-Extended and Applebot-Extended are control tokens,
# not independent crawlers — allow them to stay in Gemini/Siri.
# ----------------------------------------------------------------

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot
Allow: /

# Applebot-Extended: control token for Apple AI training.
# Allow here = your content can improve Apple Intelligence.
# Change to Disallow: / to opt out of Apple model training
# while keeping Spotlight/Siri search inclusion.
User-agent: Applebot-Extended
Allow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: Diffbot
Allow: /

# ----------------------------------------------------------------
# CATEGORY 3: AI TRAINING CRAWLERS
# These bots train generative models.
# Commerce note: blocking does NOT remove you from AI search.
# See the SEO-vs-Training section below — consider allowing
# /products/ paths if you want your product data in future
# training sets. Current config: full block.
# ----------------------------------------------------------------

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

# Bytespider has a documented history of ignoring robots.txt.
# Also add a firewall rule at your CDN as a backup layer.
User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

# Legacy/alternate Anthropic tokens — add if seen in logs
User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

# ----------------------------------------------------------------
# CATCH-ALL: Standard crawlers (Googlebot, Bingbot, etc.)
# Allow everything not covered above.
# ----------------------------------------------------------------

User-agent: *
Allow: /

# ----------------------------------------------------------------
# SITEMAP (see Agent Sitemap spoke for full sitemap strategy)
# ----------------------------------------------------------------

Sitemap: https://www.yourstore.com/sitemap.xml
Sitemap: https://www.yourstore.com/sitemap-products.xml

Partial-allow variant — open product pages to training crawlers

If you want training crawlers to learn your product catalog — normalizing your brand, products, and prices into future model knowledge — while protecting blog content, the longest-match rule makes this precise. The /products/ Allow (9 characters) beats the / Disallow (1 character), so product pages are crawled and your blog is not.

User-agent: GPTBot
Allow: /products/
Allow: /collections/
Disallow: /

User-agent: ClaudeBot
Allow: /products/
Allow: /collections/
Disallow: /

User-agent: meta-externalagent
Allow: /products/
Allow: /collections/
Disallow: /

robots.txt + llms.txt + /agents.json — three files, one discovery layer.

robots.txt is the outermost gate, but it is only one of three files that form the complete discovery layer for an agent-ready commerce site. The three specifications are additive, not competing — each answers a different question for an AI agent arriving at your domain.

agents.txt — honest status

Two parallel proposals exist under the "agents.txt" label, and they are not the same specification.

IETF Internet-Draft draft-srijal-agents-policy-00 (filed October 2025 by Srijal Dutta): A strict plaintext policy file at /agents.txt with mandatory SHA-256 hash verification of the file's canonical content. Files with missing or mismatched hashes are treated as fully restrictive (all access denied). The draft is an individual submission — not IETF-endorsed, no assigned working group, and set to expire April 2026. Zero crawlers currently implement it. (Status volatile; re-verify before launch.)

agent-manifest.txt (originally agents.txt, renamed March 2026 by Jasper van Veen): A richer capability manifest covering action permissions, API/MCP server discovery, agent identity tiers, and authentication methods. Currently at Draft v0.3.0, licensed CC BY 4.0, hosted at github.com/jaspervanveen/agents-txt. No vendor implementation. (Status volatile; re-verify before launch.)

llms.txt — brief frame for commerce

Jeremy Howard published the llms.txt proposal in September 2024 at llmstxt.org. The proposal asks sites to place a Markdown file at /llms.txt that provides LLM-friendly content: an H1 title, a blockquote summary, and organized links to key pages, optionally with .md versions of those pages for clean parsing. Deep coverage of llms.txt structure, hosting, and verification belongs in the Agent-Readable Sitemap spoke. The one robots.txt implication: make sure your robots.txt does not block AI indexers from reading /llms.txt itself — it would be self-defeating to publish an AI-friendly content guide and then block the bots that would use it.

Why commerce should not blanket-block training crawlers.

News publishers block training crawlers because their product is the article — letting GPTBot train on it means OpenAI can summarize the article without the reader visiting the publisher's site, and revenue evaporates. Commerce stores sell physical or digital goods. The product is the SKU. Training data has a fundamentally different effect on a commerce site.

The commerce-specific rule of thumb

Allow AGENT INDEXERS unconditionally (they surface you in AI answers). Allow AGENT BUYERS unconditionally (they complete purchases on users' behalf). Make a deliberate choice on training crawlers — not a reflexive block. If your product descriptions are generic (standard SKUs, commodity items), blocking training crawlers costs more than it saves. If your product copy is a genuine differentiator, a selective block using the partial-allow variant in §5 is defensible.

Confirming your rules work — curl, IP allowlists, log analysis.

Writing a robots.txt file is half the job. Verification confirms that the file is served correctly, that your server is not blocking bots at the CDN or firewall layer before they reach robots.txt, and that the right bot user-agent tokens are appearing in your logs on the right paths.

curl Tests

Simulate a specific bot's visit to verify robots.txt behavior. Note: curl -A spoofs the user-agent string in the HTTP request. The response reflects your server's behavior to that user-agent, but robots.txt compliance is evaluated by the bot — use this to verify server-side blocking, not bot obedience.

# Fetch your robots.txt — confirm it's served as text/plain
curl https://www.yourstore.com/robots.txt

# Test whether GPTBot is blocked from your homepage
curl -A "GPTBot" -I https://www.yourstore.com/

# Test whether ChatGPT-User is allowed to a product page
curl -A "ChatGPT-User" https://www.yourstore.com/products/widget-pro

# Test whether OAI-SearchBot is allowed to your sitemap
curl -A "OAI-SearchBot" https://www.yourstore.com/sitemap.xml

# Verify Bytespider gets a 200 or 403 depending on your config
curl -A "Bytespider" -I https://www.yourstore.com/products/

Vendor IP Allowlists — Verifying Bot Identity

User-agent strings can be spoofed. Major vendors publish IP ranges for reverse DNS verification. Use these to confirm that traffic claiming to be a legitimate bot is actually from that vendor.

# Step 1: Reverse DNS lookup of the suspicious IP
host <IP-FROM-LOG>
# Should resolve to *.googlebot.com or *.google.com

# Step 2: Forward-verify — resolved hostname should map back to original IP
host <resolved-hostname>
# Result should = original IP

Google Search Console

The robots.txt report in Google Search Console shows parse errors and which rules Google applied. The URL Inspection tool shows whether a specific URL is blocked by robots.txt. Submit your robots.txt for immediate re-parse after any change.

Cloudflare AI Bot Controls

Cloudflare's managed robots.txt (Security → Bots → Configure Bot Fight Mode → Instruct bot traffic with robots.txt) automatically prepends directives blocking known AI training crawlers. Useful as a base layer, but generates a blanket block — review the generated output carefully to ensure it does not block your AGENT BUYER and AGENT INDEXER categories. The Cloudflare-generated directives may include tokens that overlap with indexers depending on the version of the managed list. (Re-verify Cloudflare directive set before launch.)

Log Analysis Pattern

Parse your access logs for AI bot activity. Most declared AI bots include a recognizable substring in the user-agent header. Run this grep against your Nginx or Apache access log:

grep -iE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Amazonbot|Amzn-SearchBot|Amzn-User|meta-externalagent|Bytespider|CCBot|Applebot|Google-Extended|Diffbot" /var/log/nginx/access.log

Pipe to a frequency count by user-agent token to see which bots dominate your traffic:

grep -iE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Amazonbot|Amzn-SearchBot|Amzn-User|meta-externalagent|Bytespider|CCBot|Applebot|Diffbot" /var/log/nginx/access.log \
  | awk '{print $12}' \
  | sort | uniq -c | sort -rn \
  | head -20

Every layer of the AgentMall guide — cross-linked.

robots.txt is the outermost gate of a multi-layer agent-readiness stack. The three other discovery spokes, the platform retrofit guides, and the core protocol spokes are all linked below. Each one builds on top of what you configure here.

Discovery Siblings — the other three corners

Platform Retrofit Guides — your specific stack

Core Protocol Spokes

• Product Data — Schema.org 20-field MVP — the structured data layer agents read once they've found your page
• API Endpoint — 8-endpoint agent commerce surface — the transactional API layer
• MCP — Model Context Protocol — the tool-negotiation layer between agents and your store
• UCP — Universal Commerce Protocol — the agent-native checkout state machine
• The Agent Commerce Market — TAM, channel mix, and sizing the opportunity
• Picks & Shovels — cross-platform tools, validators, and monitoring
• AgentMall Roadmap — the pillar page tying all four layers together

Eight robots.txt errors that cut you off from AI commerce.

1. Wrong user-agent token for ChatGPT browsing

Using ChatGPT as the user-agent token — it is not valid. The bot that fetches on a user's behalf uses the token ChatGPT-User. The training crawler uses GPTBot. These are two entirely separate bots with separate tokens, separate functions, and separate robots.txt rules. A rule written for ChatGPT matches nothing. A rule written for GPTBot has zero effect on ChatGPT-User.

2. Blanket-blocking all AI bots and losing buyer-agent traffic

Adding a User-agent: * / Disallow: / block or applying Cloudflare's managed robots.txt without review catches all AI bots including Agent Buyers and Agent Indexers. The correct structure: explicit Allow: / rules for all AGENT BUYERS and AGENT INDEXERS by their specific user-agent tokens, then explicit Disallow: / for training crawlers you want to exclude, then User-agent: * / Allow: / for everything else. The specific rules override the wildcard for those bots.

3. Assuming blocking GPTBot stops ChatGPT from finding you

GPTBot trains the model weights. OAI-SearchBot builds the search index ChatGPT answers draw from. ChatGPT-User fetches your page live during a shopping session. Blocking only GPTBot leaves OAI-SearchBot and ChatGPT-User fully active — your products still surface in ChatGPT shopping. If you want to block all OpenAI access, you need separate Disallow: / rules for all three tokens independently.

4. Using meta robots noai/noimageai tags as the primary control

The HTML meta directives <meta name="robots" content="noai, noimageai"> are not part of RFC 9309 and are not universally respected. Meta (the company) explicitly uses robots.txt, not NoAI tags. Anthropic, Google, and OpenAI do not document support for these tags. Use robots.txt user-agent blocks as your primary control. HTML meta tags may evolve, but robots.txt is the standard today.

5. Blocking by IP instead of user-agent token

Blocking a vendor's published IP ranges at the firewall prevents the bot from reading your robots.txt at all — Anthropic explicitly warns that IP blocking "may impede Anthropic's ability to read robots.txt." A bot that cannot read robots.txt may treat your site as unconstrained. Use IP blocking only as a supplementary layer for confirmed bad-actor crawlers (Bytespider with documented compliance issues) after setting the correct user-agent token rule in robots.txt first.

6. Forgetting the Sitemap directive

Without a Sitemap: directive, AI indexers may still crawl your site by following links — but they start discovery from whatever links they already know. The Sitemap directive is not part of the formal RFC 9309 user-agent/rule grammar, but it is universally supported as an extension. Add it at the end of your robots.txt file to give every crawler — AI and traditional — a systematic URL inventory. For the full sitemap strategy, see the Agent-Readable Sitemap spoke.

7. Case-sensitive path errors in Disallow rules

RFC 9309 specifies that paths in robots.txt are case-sensitive. Disallow: /Private/ does not match /private/. Disallow: /Admin/ does not match /admin/. Verify the actual URL casing your server uses before writing Disallow rules for sensitive paths — especially on case-insensitive file systems (most Windows servers) where the URL paths may look equivalent but robots.txt matching is case-sensitive regardless.

8. Misconfiguring Google-Extended and losing Gemini grounding

Google-Extended does not send HTTP requests. It is a robots.txt control token that instructs Google's existing crawlers whether already-crawled content may be used for Gemini model training and grounding. Disallowing it does not protect you from Googlebot, does not affect Search rankings, and does not stop Google from knowing your content exists. It only removes your content from Gemini Apps responses and Vertex AI grounding. If you want to stay in Gemini while keeping Google Search rankings, keep Google-Extended Allowed.

Frequently asked questions.

Configuring robots.txt for agent commerce, in five steps.

Each step mirrors the HowTo JSON-LD at the top of this page word for word. Execute in order — the output of each step is the input to the next.

Step 1 — Inventory bots in your logs

Pull the last 30 days of access logs and extract all user-agent strings that include known AI bot tokens. Use the grep pattern from the Verification section. Create a spreadsheet with columns: token, request count, paths accessed, source IPs. This baseline tells you which bots are actually hitting your site today and whether any paths you expected to be accessible are getting 403s or redirects.

Step 2 — Classify each bot into buyer / indexer / training

Using the Bot Directory table and the taxonomy summary, assign each active bot to one of the three categories. Flag any unrecognized tokens for manual research — look them up against the vendor's official documentation, not community blocklists. Pay attention to bots where the classification is ambiguous (e.g., meta-externalagent is described by Meta as covering both AI indexing and training).

Step 3 — Draft your allow/block policy

For each category, apply the commerce-default rules from this document: Allow all buyers, Allow all indexers, make a deliberate choice on training crawlers (all-block, selective path allow, or full allow). Document the business rationale for each training crawler decision — you will revisit this as AI commerce matures. Decide whether you want Google-Extended and Applebot-Extended Allowed (your content may improve future Gemini/Apple Intelligence responses) or Disallowed (your content stays out of those training pipelines but you remain in Google Search and Spotlight).

Step 4 — Deploy robots.txt + cross-references

Replace or update your robots.txt with the complete runnable file from the Reference robots.txt section, adjusted for your actual paths. Add the Sitemap: directive with your product sitemap URL. Publish /llms.txt linking to your key product pages. Publish /agents.json with your store's API surface. Verify robots.txt is accessible at exactly https://www.yourstore.com/robots.txt with Content-Type: text/plain and UTF-8 encoding. Submit the updated robots.txt in Google Search Console for immediate re-parse.

Step 5 — Verify with curl, logs, and vendor tools

Run curl tests for each major bot on representative URLs: your homepage, a product page, a collection page, and any admin or private paths that should be blocked. Check Google Search Console's robots.txt report for parse errors. Pull logs 48–72 hours after deployment to confirm that AGENT BUYER tokens are appearing on product pages and that training crawler tokens are not. Set a calendar reminder to re-check vendor bot documentation quarterly — user-agent strings and version numbers change, and new bots are introduced without notice.

Next stops in the Discovery quadrant.