|
4 | 4 |
|
5 | 5 | Check whether user-agent is an AI crawlers |
6 | 6 |
|
7 | | -```csv:no-line-numbers |
8 | | -# Current list of AI Crawlers: |
9 | | -AI2_BOT, AMAZON_BOT, ANTHROPIC_AI, ANTHROPIC_CLAUDE_BOT, |
10 | | -ANTHROPIC_CLAUDE_SEARCHBOT, ANTHROPIC_CLAUDE_WEB, APPLE_BOT, |
11 | | -APPLE_BOT_EXTENDED, BRAVE_BOT, BYTEDANCE_BYTESPIDER, BYTEDANCE_TIKTOKSPIDER, |
12 | | -COHERE_TRAINING_DATA_CRAWLER, COMMON_CRAWL_CCBOT, COVEO_BOT, DATAFORSEO_BOT, |
13 | | -DEEPSEEK_BOT, DIFFBOT, GOOGLE_EXTENDED, GOOGLE_OTHER, GOOGLE_OTHER_IMAGE, |
14 | | -GOOGLE_OTHER_VIDEO, GOOGLE_CLOUDVERTEXBOT, HIVE_IMAGESIFTBOT, |
15 | | -HUAWEI_PETALBOT, HUAWEI_PANGUBOT, HUGGINGFACE_BOT, KANGAROO_BOT, |
16 | | -FIRECRAWL_AGENT, META_FACEBOOKBOT, META_EXTERNALAGENT, OPENAI_GPTBOT, |
17 | | -OPENAI_SEARCH_BOT, PERPLEXITY_BOT, REPLICATE_BOT, RUNPOD_BOT, |
18 | | -SB_INTUITIONS_BOT, SEMRUSH_BOT_CONTENTSHAKE, TIMPI_BOT, TOGETHER_BOT, |
19 | | -HUNTER_VELENPUBLICWEBCRAWLER, VERCEL_V0BOT, WEBZIO_OMGILI, |
20 | | -WEBZIO_OMGILI_BOT, WEBZIO_EXTENDED, XAI_BOT, YOU_BOT, ZHIPU_CHATGLM_SPIDER |
21 | | -``` |
| 7 | +| **Provider** | **UA Token** | |
| 8 | +| ------------- | --------------------------------------------------------------------------------------------------- | |
| 9 | +| AI2 | `AI2Bot` | |
| 10 | +| Amazon | `Amazonbot` | |
| 11 | +| Anthropic | `anthropic-ai`, `ClaudeBot`, `Claude-SearchBot`, `Claude-Web` | |
| 12 | +| Apple | `Applebot`, `Applebot-Extended` | |
| 13 | +| Brave | `Bravebot` | |
| 14 | +| ByteDance | `Bytespider`, `TikTokSpider` | |
| 15 | +| Cohere | `cohere-training-data-crawler` | |
| 16 | +| Common Crawl | `CCBot` | |
| 17 | +| Coveo | `Coveobot` | |
| 18 | +| DataForSEO | `DataForSeoBot` | |
| 19 | +| DeepSeek | `DeepSeekBot` | |
| 20 | +| Diffbot | `Diffbot` | |
| 21 | +| Google | `Google-Extended`, `GoogleOther`, `GoogleOther-Image`, `GoogleOther-Video`, `Google-CloudVertexBot` | |
| 22 | +| Hive AI | `ImagesiftBot` | |
| 23 | +| Huawei | `PanguBot`, `PetalBot` | |
| 24 | +| Hugging Face | `HuggingFace-Bot` | |
| 25 | +| Kangaroo | `Kangaroo Bot` | |
| 26 | +| Mendable.ai | `FirecrawlAgent` | |
| 27 | +| Meta | `FacebookBot`, `meta-externalagent` | |
| 28 | +| OpenAI | `GPTBot`, `OAI-SearchBot` | |
| 29 | +| Perplexity | `PerplexityBot` | |
| 30 | +| Replicate | `Replicate-Bot` | |
| 31 | +| RunPod | `RunPod-Bot` | |
| 32 | +| SB Intuitions | `SBIntuitionsBot` | |
| 33 | +| SEMrush | `SemrushBot-OCOB` | |
| 34 | +| Timpi | `Timpibot` | |
| 35 | +| Together AI | `Together-Bot` | |
| 36 | +| Hunter.io | `VelenPublicWebCrawler` | |
| 37 | +| Vercel | `v0bot` | |
| 38 | +| Webz.io | `omgili`, `omgilibot`, `Webzio-Extended` | |
| 39 | +| xAI | `xAI-Bot` | |
| 40 | +| You.com | `YouBot` | |
| 41 | +| Zhipu AI | `ChatGLM-Spider` | |
| 42 | + |
| 43 | + |
22 | 44 | ::: tip |
23 | | -See list of possible values for `Crawler` enum [here](/api/submodules/enums/extension#extension-browsername-crawler). |
| 45 | +See list of all possible values for `Crawler` [here](/api/submodules/extensions/crawlers). |
24 | 46 | ::: |
25 | 47 |
|
26 | 48 | ## Code Example |
|
0 commit comments