# robots.txt - verifiedpetfooddata.com
#
# /go/ is the affiliate redirect layer. It is disallowed for EXACTLY ONE agent:
# Bytespider. Every other agent may follow it. This matches the policy proven on
# verifiedsupplementdata.com (see ~/verifiedx-intel/DECISIONS.md, 2026-08-16).
#
# 2026-08-19: rewritten. The previous file carried `Disallow: /go/` in EVERY block
# but wrote it AFTER `Allow: /`. Google and Bing resolve conflicts by LONGEST-MATCH,
# so that order works for them - but order-based parsers take the FIRST match and
# read ALLOW. Verified with urllib.robotparser: every agent, including Bytespider,
# could fetch /go/. The whole file was a silent no-op.
#
# ⛔ The fix is NOT "make all those disallows work." Blocking /go/ for the OpenAI
# agents is the leading suspected cause of VSD's 2026-08-07 ChatGPT referral cliff
# (94 sessions/day -> 44, never recovered), and AI assistants convert at ~207
# affiliate clicks per 1k sessions vs 37 for everything else. They are the growth
# surface. So /go/ stays OPEN for them, exactly as it is on VSD today.
#
# Bytespider is blocked because it is measured pure cost: on VSD it followed 535
# /go/ redirects in one day - 98% of the entire crawler increase - while sending
# zero referral sessions (ByteDance has no search product pointing here).
#
# ⚠️ Disallow FIRST, then Allow, in the Bytespider block. Correct under BOTH
# longest-match and first-match semantics. Do not reorder these two lines.
# ⚠️ A named `User-agent:` block REPLACES the `*` block - it does NOT inherit - and
# a BLANK LINE ends a block. Any Disallow added to `*` must be repeated in each.

User-agent: *
Allow: /

# OpenAI - ChatGPT Search
User-agent: OAI-SearchBot
Allow: /

# OpenAI - Real-time user fetches
User-agent: ChatGPT-User
Allow: /

# OpenAI - Training/indexing
User-agent: GPTBot
Allow: /

# Anthropic - Claude Search (powers Claude web search via Brave)
User-agent: Claude-SearchBot
Allow: /

# Anthropic - Real-time user fetches
User-agent: Claude-User
Allow: /

# Anthropic - Training/indexing
User-agent: ClaudeBot
Allow: /

# Perplexity - Independent index
User-agent: PerplexityBot
Allow: /

# Google - Search index (feeds AI Overviews)
User-agent: Googlebot
Allow: /

# Google - AI training (allowing = more visibility)
User-agent: Google-Extended
Allow: /

# Microsoft - Bing + Copilot
User-agent: bingbot
Allow: /

# Apple Intelligence
User-agent: Applebot
Allow: /

# ByteDance / TikTok AI
User-agent: Bytespider
Disallow: /go/
Allow: /

# Meta AI
User-agent: Meta-ExternalAgent
Allow: /

# DuckDuckGo AI
User-agent: DuckAssistBot
Allow: /

Sitemap: https://verifiedpetfooddata.com/sitemap-index.xml