User-agent: * Allow: / # AI crawler limits User-agent: anthropic-ai Crawl-delay: 5 User-agent: ClaudeBot Crawl-delay: 5 User-agent: GPTBot Crawl-delay: 5