AI crawler directory

OAI-SearchBot

OAI-SearchBot

Surfaces websites in ChatGPT search products.

ClaudeBot

ClaudeBot

Automated web crawling by Anthropic.

View crawler →AI Assistants

Claude-SearchBot

Claude-SearchBot

Supports search results for Claude users.

Claude-User

Claude-User

User-initiated retrieval in Claude.

Google-Extended

Google-Extended

Controls use of crawled content for Gemini models and grounding.

Googlebot

Googlebot

Google web search crawler.

Googlebot Image

Googlebot-Image

Google image search crawler.

Bingbot

bingbot

Microsoft Bing search crawler.

Applebot

Applebot

Supports Apple search products.

Applebot-Extended

Applebot-Extended

Controls use of crawled content for Apple generative models.

View crawler →AI Assistants

PerplexityBot

PerplexityBot

Indexes content for Perplexity search.

Perplexity-User

Perplexity-User

User-requested retrieval by Perplexity.

View crawler →Archive/Data Crawlers

CCBot

CCBot

Builds the Common Crawl public web corpus.

Bytespider

Bytespider

ByteDance web crawler; published purpose details are limited.

View crawler →Social Crawlers

Amazonbot

Amazonbot

Supports Amazon services including search and assistant features.

FacebookBot

FacebookBot

Fetches content for Meta link and AI features.

View crawler →Social Crawlers

Twitterbot

Twitterbot

Generates link previews on X.

View crawler →Social Crawlers

LinkedInBot

LinkedInBot

Generates LinkedIn link previews.

DuckDuckBot

DuckDuckBot

DuckDuckGo search crawler.

YandexBot

YandexBot

Yandex search crawler.