AI crawlers, search bots, and training agents.
A compact reference of the crawler identities Faurya checks: what they are for, how they identify themselves, and where their verification sources live.
52 crawlers found
AI answers
User-triggered / 15 crawlers
Anthropic
Claude-User
Claude-User is tracked as a user-triggered fetcher from Anthropic.
Google-agent
Google-agent is tracked as a user-triggered fetcher from Google.
Google-NotebookLLM
Google-NotebookLLM is tracked as a user-triggered fetcher from Google.
Google-Read-Aloud
Google-Read-Aloud is tracked as a user-triggered fetcher from Google.
GoogleAgent
GoogleAgent is tracked as a user-triggered fetcher from Google.
Preplexity
Perplexity-User
Perplexity-User is tracked as a user-triggered fetcher from Preplexity.
Mistral
MistralAI-User
MistralAI-User is tracked as a user-triggered fetcher from Mistral.
Microsoft
Copilot
Copilot is tracked as a user-triggered fetcher from Microsoft.
Amazon
Amzn-User
Amzn-User is tracked as a user-triggered fetcher from Amazon.
DuckDuckGo
DuckAssistBot
DuckAssistBot is tracked as a user-triggered fetcher from DuckDuckGo.
Meta
meta-externalfetcher
meta-externalfetcher is tracked as a user-triggered fetcher from Meta.
xAI
xAI-SearchBot
xAI-SearchBot is tracked as a user-triggered fetcher from xAI.
xAI
Grok-DeepSearch
Grok-DeepSearch is tracked as a user-triggered fetcher from xAI.
Moonshot AI
Kimi-User
Kimi-User is tracked as a user-triggered fetcher from Moonshot AI.
Alibaba
Qwen-User
Qwen-User is tracked as a user-triggered fetcher from Alibaba.
Search indexes
Discovery / 13 crawlers
Anthropic
Claude-SearchBot
Claude-SearchBot is tracked as a search or answer-index crawler from Anthropic.
Google-InspectionTool
Google-InspectionTool is tracked as a search or answer-index crawler from Google.
GoogleOther
GoogleOther is tracked as a search or answer-index crawler from Google.
Googlebot
Googlebot is tracked as a search or answer-index crawler from Google.
Preplexity
PerplexityBot
PerplexityBot is tracked as a search or answer-index crawler from Preplexity.
Mistral
MistralAI-Index
MistralAI-Index is tracked as a search or answer-index crawler from Mistral.
Microsoft
bingbot
bingbot is tracked as a search or answer-index crawler from Microsoft.
Microsoft
msnbot
msnbot is tracked as a search or answer-index crawler from Microsoft.
Amazon
Amzn-SearchBot
Amzn-SearchBot is tracked as a search or answer-index crawler from Amazon.
Moonshot AI
Kimi-SearchBot
Kimi-SearchBot is tracked as a search or answer-index crawler from Moonshot AI.
Apple
Applebot
Applebot is tracked as a search or answer-index crawler from Apple.
ByteDance
TikTokSpider
TikTokSpider is tracked as a search or answer-index crawler from ByteDance.
Baidu
Baiduspider
Baiduspider is tracked as a search or answer-index crawler from Baidu.
Training crawlers
Model data / 13 crawlers
Anthropic
ClaudeBot
ClaudeBot is tracked as a public-content or model-data crawler from Anthropic.
Amazon
Amazonbot
Amazonbot is tracked as a public-content or model-data crawler from Amazon.
Meta
meta-externalagent
meta-externalagent is tracked as a public-content or model-data crawler from Meta.
Moonshot AI
KimiBot
KimiBot is tracked as a public-content or model-data crawler from Moonshot AI.
Alibaba
QwenBot
QwenBot is tracked as a public-content or model-data crawler from Alibaba.
Apple
Applebot-Extended
Applebot-Extended is tracked as a public-content or model-data crawler from Apple.
ByteDance
Bytespider
Bytespider is tracked as a public-content or model-data crawler from ByteDance.
Baidu
ERNIEBot
ERNIEBot is tracked as a public-content or model-data crawler from Baidu.
AllenAI
AI2Bot
AI2Bot is tracked as a public-content or model-data crawler from AllenAI.
DeepSeek
DeepSeekBot
DeepSeekBot is tracked as a public-content or model-data crawler from DeepSeek.
Cohere
cohere-training-data-crawler
cohere-training-data-crawler is tracked as a public-content or model-data crawler from Cohere.
Cohere
cohere-ai
cohere-ai is tracked as a public-content or model-data crawler from Cohere.
Common Crawl
CCBot
CCBot is tracked as a public-content or model-data crawler from Common Crawl.
Other AI bots
AI crawler / 11 crawlers
Google-CloudVertexBot
Google-CloudVertexBot is tracked as another AI bot from Google.
Meta
FacebookBot
FacebookBot is tracked as another AI bot from Meta.
xAI
GrokBot
GrokBot is tracked as another AI bot from xAI.
xAI
xAI-Bot
xAI-Bot is tracked as another AI bot from xAI.
xAI
xAI-Grok
xAI-Grok is tracked as another AI bot from xAI.
xAI
xAI-Web-Crawler
xAI-Web-Crawler is tracked as another AI bot from xAI.
xAI
Grok
Grok is tracked as another AI bot from xAI.
Alibaba
AliyunBot
AliyunBot is tracked as another AI bot from Alibaba.
Alibaba
TongyiBot
TongyiBot is tracked as another AI bot from Alibaba.
ByteDance
Doubaobot
Doubaobot is tracked as another AI bot from ByteDance.
Baidu
YiyanBot
YiyanBot is tracked as another AI bot from Baidu.