# Default robots file version:2 User-agent: * Disallow: /calendar/action* Disallow: /events/action* Allow: /*.css Allow: /*.js Disallow: /*? Crawl-delay: 3 Sitemap: https://whyinstitute.com/sitemap.xml # ============================ # ANTHROPIC (Claude) # ============================ User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / # ============================ # OPENAI (ChatGPT) # ============================ User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / # ============================ # GOOGLE (Gemini) # Note: do NOT block Googlebot itself — that removes you from Google Search # ============================ User-agent: Google-Extended Disallow: / User-agent: GoogleOther Disallow: / # ============================ # PERPLEXITY # ============================ User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # ============================ # META (Facebook/Instagram AI) # ============================ User-agent: Meta-ExternalAgent Disallow: / User-agent: FacebookBot Disallow: / # ============================ # APPLE # ============================ User-agent: Applebot-Extended Disallow: / # ============================ # AMAZON (Alexa / Rufus) # ============================ User-agent: Amazonbot Allow: / # ============================ # BYTEDANCE # ============================ User-agent: Bytespider Disallow: / # ============================ # COMMON CRAWL (feeds many models' training sets) # ============================ User-agent: CCBot Disallow: / # ============================ # COHERE # ============================ User-agent: cohere-ai Disallow: / # ============================ # DUCKDUCKGO AI # ============================ User-agent: DuckAssistBot Allow: /