Search Engines
Essential bots that power search visibility - typically kept allowed
Social Media
Bots that generate link previews for social platforms - usually allowed
AI Search
AI-powered search and user-triggered browsing - mixed compliance
AI Training
Bots that collect data for AI model training - often blocked
SEO Tools
SEO analysis and backlink checking tools - depends on usage
Generated Bot Blocking Code
Currently blocking 0 bots
# No bots selected for blocking
| Bot Name | What It Does |
|---|---|
| Googlebot | Google's primary crawler for web indexing |
| Googlebot-Image | Indexes images for Google Images |
| Googlebot-Video | Indexes videos for Google Video search |
| Googlebot-News | Indexes content for Google News |
| Google-InspectionTool | Google Search Console testing and inspection |
| Storebot-Google | Indexes products for Google Shopping |
| bingbot | Microsoft's crawler for Bing search indexing |
| DuckDuckBot | Privacy-focused search engine crawler |
| YandexBot | Russian search engine with 65% market share in Russia |
| Baiduspider | Chinese search engine crawler (often aggressive) |
| Applebot | Apple's crawler for Siri and Spotlight |
| SeznamBot | Czech search engine with 15% market share |
| Qwantify | French/EU privacy-focused search engine |
| Sogou web spider | Chinese search engine (poor robots.txt compliance) |
| Yeti | Naver's Korean search engine (47% Korean market) |
| facebookexternalhit | Generates link previews for Facebook (ignores robots.txt for user shares) |
| Twitterbot | Generates link previews for X/Twitter |
| LinkedInBot | Creates link previews for LinkedIn posts |
| Pinterestbot | Indexes content for Pinterest pins |
| Discordbot | Creates link embeds for Discord messages |
| Slackbot | Generates link unfurling for Slack messages |
| ChatGPT-User | User-triggered browsing from ChatGPT |
| OAI-SearchBot | OpenAI's SearchGPT indexing bot |
| Claude-User | User-triggered content retrieval from Claude |
| PerplexityBot | Perplexity AI search engine (ignores robots.txt) |
| Perplexity-User | User-triggered searches from Perplexity |
| Applebot-Extended | Apple Intelligence AI search features |
| cohere-ai | User-triggered queries for Cohere AI |
| Meta-ExternalFetcher | Meta's user-triggered content fetching |
| GPTBot | OpenAI's crawler for training GPT models |
| ClaudeBot | Anthropic's crawler for training Claude models |
| Google-Extended | Google's Gemini AI training (separate from search) |
| CCBot | Common Crawl open dataset for research |
| Bytespider | ByteDance/TikTok crawler (extremely aggressive, ignores robots.txt) |
| Meta-ExternalAgent | Meta's AI training crawler (poor compliance) |
| FacebookBot | Meta's AI data scraping bot |
| cohere-training-data-crawler | Cohere's training data collection |
| Amazonbot | Amazon's product and AI data crawler |
| Qwen | Alibaba's AI training crawler |
| AhrefsBot | Backlink analysis and SEO metrics |
| SemrushBot | SEO analysis platform (high bandwidth usage) |
| MJ12bot | Majestic's distributed backlink analysis |
| DotBot | Moz's link index building |
| Screaming Frog SEO Spider | Desktop-based technical SEO crawler |
| BLEXBot | WebMeUp's backlink checker |
| SEOkicks-Robot | European-focused SEO analysis tool |
| rogerbot | Moz Pro site audits |