Glossary · AI search and AEO
What are AI crawlers and their user agents?
Also known as: AI bots, AI user agents, LLM crawlers
AI crawler (user agents): An AI crawler is a bot that AI companies use to fetch web pages, and its user agent is the name it identifies itself by in requests and in robots.txt rules. Providers typically run separate agents for model training, search indexing and fetching a page when a user asks.
Why does it matter?
Blocking the wrong user agent can quietly remove you from AI search answers. Many site owners blocked AI bots wholesale to stop training use, not realising the same rule could also stop search crawlers that cite pages.
The split between purposes lets you choose. Providers such as OpenAI and Anthropic document different agents for training and for search, so you can generally opt out of one while allowing the other.
How do you act on it?
- Read your robots.txt for broad disallow rules and for named AI agents, and confirm each rule matches your intent.
- Check your CDN, firewall or bot-protection settings, which can challenge or block AI crawlers even when robots.txt allows them.
- Look up each provider's current crawler documentation before editing; names and behaviour change.
- Review server logs occasionally to see which AI agents actually visit.
Frequently asked questions
Which AI user agents are there?
Examples documented by providers include OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity's PerplexityBot and Perplexity-User. Google uses the Google-Extended token to control certain AI uses. Lists change, so check each provider's documentation.
If I block training crawlers, will I disappear from AI answers?
Not necessarily. Where a provider separates training and search crawlers, blocking the training one generally doesn't block the search one. The exact effect depends on the provider, so confirm in their current docs.