AI Crawler
A bot that fetches web content for training or for grounding live AI answers.
Distinguishing crawler purposes is now an operational decision. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others have different roles, and blocking them has different consequences: blocking a training crawler affects future models, while blocking a retrieval crawler can remove you from answers today. Many travel sites block both by accident through a copied robots.txt.
Related terms
Related indicators
Every figure carries its publisher, sample and field period.