# Welcome AI Agents and Crawlers! # CatalogAPI is designed to be AI-agent friendly # Learn more at: https://api.catalogapi.com/developers/ai-agents # # Content Signals (https://contentsignals.org/) state how we want this content # used: # search building a search index and linking back to us # ai-input reading a page to answer a question at the time it is asked # ai-train training or fine-tuning a model # All three are yes. Our docs are meant to be found, read, and learned from. # Agents that fetch live get the freshest version: see /.well-known/api-catalog. # # The signal and the Disallow lines are repeated in every group on purpose. A # crawler reads only the group that matches it best and ignores the others, so # anything declared once under "User-agent: *" would never reach the named # crawlers below. Keep the Disallow lines in step across groups when editing. # AI Assistant Crawlers - Full Access # # Each operator runs separate agents for training, for answering a live user # question, and for building a search index. They are named individually # because blocking or allowing one says nothing about the others. # OpenAI: GPTBot trains, OAI-SearchBot builds the ChatGPT search index, # ChatGPT-User fetches a page because someone asked. OAI-AdsBot is omitted; it # validates ad landing pages and we run no ads. User-agent: GPTBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 1 User-agent: OAI-SearchBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 1 User-agent: ChatGPT-User Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 1 # Anthropic: ClaudeBot trains, Claude-SearchBot indexes, Claude-User fetches # for a live question. Claude-Web was the old single token and is retired, so # naming it here would reach nothing. User-agent: ClaudeBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 1 User-agent: Claude-SearchBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 1 User-agent: Claude-User Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 1 # Perplexity: PerplexityBot indexes. Perplexity-User fetches for a live # question and their docs say it generally ignores robots.txt, so this group # records our position rather than controlling it. User-agent: PerplexityBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 1 User-agent: Perplexity-User Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ User-agent: CCBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 2 # Search Engines - Full Access User-agent: Googlebot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 1 User-agent: Bingbot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 1 # AI Training Permission Tokens # # Neither of these fetches anything. They are the switches Google and Apple # read to decide whether content their normal crawlers already took may train # a model. Allowing them is the same answer ai-train=yes gives above, in the # form those two operators actually act on. User-agent: Google-Extended Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / User-agent: Applebot-Extended Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # Documentation Crawlers User-agent: Applebot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: /api/docs Allow: /developers/ Allow: /faq Disallow: /users/ Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ # General Crawlers User-agent: * Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Allow: /api/docs Allow: /.well-known/api-catalog Allow: /.well-known/agent-skills/ Allow: /.well-known/agent-skills/index.json Allow: /.well-known/mcp/server-card.json Allow: /.well-known/oauth-protected-resource Allow: /auth.md Allow: /llms.txt Allow: /api/v2/.well-known/ai-context Allow: /developers/ Allow: /faq Disallow: /users/settings Disallow: /admin/ Disallow: /dev/ Disallow: /browse/ Crawl-delay: 2 # Sitemap for better discovery Sitemap: https://catalogapi.com/sitemap.xml # AI Context Endpoint - Machine-readable API information # GET https://api.catalogapi.com/api/v2/.well-known/ai-context # # API Catalog (RFC 9727) - Links to the OpenAPI description and docs # GET https://api.catalogapi.com/.well-known/api-catalog # Also advertised in a Link header on every public page. # # Agent Skills - Task guides for coding agents, one SKILL.md each # GET https://api.catalogapi.com/.well-known/agent-skills/index.json # # llms.txt - Curated map of this site, for a model reading it cold # GET https://api.catalogapi.com/llms.txt # # MCP Server - Ask an assistant about us directly, no API key needed # GET https://api.catalogapi.com/.well-known/mcp/server-card.json # POST https://api.catalogapi.com/mcp (streamable-http, JSON-RPC) # The card is listed above; /mcp itself is not, because it speaks JSON-RPC over # POST and answers a crawler's GET with 406. Read the card instead. # Registered as com.catalogapi/catalogapi in the official MCP registry.