# Allow all good bots User-agent: * Allow: / # Block pages that waste crawl budget (no SEO value) Disallow: /api/ Disallow: /thankyou Disallow: /thank-you Disallow: /thanks Disallow: /hrm-software/thank-you Disallow: /hrm-software/thankyou Disallow: /attendance-app/thank-you Disallow: /attendance-app/thankyou Disallow: /crm-software/thank-you Disallow: /attendance-app/thank-you/ Disallow: /form-India-Soft # ── LLM & AI Crawlers — explicitly allowed ── # OpenAI (ChatGPT, GPT-4) User-agent: GPTBot Allow: / # OpenAI content crawler User-agent: OAI-SearchBot Allow: / # Anthropic (Claude) User-agent: ClaudeBot Allow: / # Anthropic legacy User-agent: anthropic-ai Allow: / # Perplexity AI User-agent: PerplexityBot Allow: / # Google AI (Gemini, AI Overviews, Bard) User-agent: Google-Extended Allow: / # Meta AI (Llama) User-agent: FacebookBot Allow: / # Apple User-agent: Applebot Allow: / # Amazon Alexa User-agent: Amazonbot Allow: / # Common Crawl (used to train many LLMs) User-agent: CCBot Allow: / # Cohere User-agent: cohere-ai Allow: / # ── SEO bots — rate limited ── User-agent: baiduspider Disallow: / # Sitemap location Sitemap: https://ubitechsolutions.com/sitemap.xml # LLMs.txt: https://ubitechsolutions.com/llms.txt