What Is a Search Bot? Search Engine and AI Crawlers in 2026
A search bot visits pages so a search engine or AI assistant can find them. See which bots matter in 2026, what each one does and which to allow in robots.txt.
Key Takeaways
5- A search bot is a program that visits web pages so a search engine or AI assistant can find, read and show them. Googlebot and Bingbot are the best-known examples.
- AI companies run several bots with different jobs: one for search, one for model training and one that fetches a page when a user asks. Each needs its own robots.txt decision.
- Blocking a search bot keeps you out of that product’s answers. Blocking a training bot does not. Mixing the two up is the most common mistake.
- Google-Extended and Applebot-Extended are not crawlers at all. They are robots.txt tokens that control AI training, and they do not affect search results.
- A user-agent can be faked. Google, OpenAI, Anthropic, Perplexity, Apple and others publish IP ranges so you can check that a visit is real.
A search bot, also called a crawler or spider, is a program that automatically visits web pages so that a search engine or an AI assistant can find them, read them and show them to people. When Google shows your page in its results, or ChatGPT cites it in an answer, a bot fetched that page first.
For years “search bot” mostly meant Googlebot and Bingbot. In 2026 the list is longer. OpenAI, Anthropic, Perplexity, Apple, Meta and Amazon all send their own bots, and most of them send more than one. Knowing which is which decides whether your pages can appear in AI answers.
How does a search bot work?
A search bot does three things over and over.
- Discover. It finds URLs from links, sitemaps and pages it has already seen.
- Fetch. It requests the page from your server, much like a browser would, and identifies itself with a name in the user-agent header.
- Hand over. It passes what it found to the system behind it. For a search engine, that is the index. For an AI assistant, it may be a search index, a training dataset, or a single answer someone is waiting for right now.
Before fetching, well-behaved bots read your robots.txt file and skip what you ask them to skip. That file is the main way to tell bots what they may visit.
What kinds of bots visit a website?
Not every bot does the same job, and that matters more than the brand name on it.
| Type | What it does | Example |
|---|---|---|
| Search engine crawler | Builds the index behind a search engine | Googlebot, bingbot |
| AI search crawler | Finds pages that an AI assistant can cite in answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Training crawler | Collects pages that may be used to train AI models | GPTBot, ClaudeBot, CCBot |
| User-triggered fetcher | Opens a page because a user asked the assistant to | ChatGPT-User, Claude-User, Perplexity-User |
| Control token | Not a crawler; a name you use in robots.txt to opt out of AI training | Google-Extended, Applebot-Extended |
The practical point: blocking an AI search crawler removes you from that assistant’s answers, while blocking a training crawler does not. The two are separate decisions.
Search engine bots
These are the crawlers behind classic search results, and in Google’s and Microsoft’s case behind their AI answers too.
| User-agent | Company | What it is for |
|---|---|---|
Googlebot | Google Search, including AI Overviews and AI Mode, Discover, Images and News | |
bingbot | Microsoft | Bing search. Copilot uses the same crawling and index |
Applebot | Apple | Spotlight, Siri and Safari suggestions |
DuckDuckBot | DuckDuckGo | DuckDuckGo search results |
YandexBot | Yandex | Yandex search index |
Google says AI Overviews and AI Mode use the same Googlebot crawling as the rest of Search, so there is no separate bot to allow for them. Google Search Central: AI features. Microsoft says Bing and Copilot share the same crawling, indexing and ranking foundation. Bing Webmaster Guidelines.
AI bots: the list for 2026
Every company below documents its bots on an official page, linked in the last column. Bot names and rules change, so treat those pages as the source of truth.
| User-agent | Company | Job | Follows robots.txt | Source |
|---|---|---|---|---|
OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search; not used for training | Yes | OpenAI |
GPTBot | OpenAI | Collects content for training OpenAI models | Yes | OpenAI |
ChatGPT-User | OpenAI | Opens a page when a ChatGPT user or GPT asks for it | Not necessarily | OpenAI |
Claude-SearchBot | Anthropic | Search index for Claude | Yes | Anthropic |
ClaudeBot | Anthropic | Collects content for model training | Yes | Anthropic |
Claude-User | Anthropic | Fetches a page a Claude user asked for | Controllable via robots.txt | Anthropic |
PerplexityBot | Perplexity | Finds pages to show and link in Perplexity; not used for training | Yes | Perplexity |
Perplexity-User | Perplexity | Fetches a page a user asked about | Generally no | Perplexity |
Google-Extended | Token, not a crawler: opts out of Gemini training and grounding; no effect on Search | n/a | ||
Applebot-Extended | Apple | Token, not a crawler: opts out of Apple AI training; no effect on search | n/a | Apple |
meta-externalagent | Meta | Training and indexing for Meta AI products | Yes | Meta |
meta-externalfetcher | Meta | Fetches a page for a user or an agent task | May not | Meta |
Amazonbot | Amazon | Amazon products and services; may train Amazon AI models | Yes | Amazon |
Amzn-SearchBot | Amazon | Search experiences such as Alexa; not used for training | Yes | Amazon |
DuckAssistBot | DuckDuckGo | Fetches pages for cited AI answers; not used for training | Yes | DuckDuckGo |
CCBot | Common Crawl | Builds the open Common Crawl dataset many AI models are trained on | Yes | Common Crawl |
Two things in this table surprise people. First, user-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt, because the companies treat them like a person clicking a link. Second, Google-Extended and Applebot-Extended never visit your site. They are names you write in robots.txt, and the regular Googlebot and Applebot read them.
You may also see Bytespider, which is widely attributed to ByteDance. We could not find an official ByteDance page describing it, so we have left it out of the table.
Is that bot really who it says it is?
Anyone can put “Googlebot” in a user-agent header. Scrapers do it all the time. Before you trust a visit, or before you block one, check where it came from.
- Published IP lists. Google, OpenAI, Anthropic, Perplexity, Apple, Common Crawl, Amazon and DuckDuckGo publish the IP ranges their bots use. Google explains how to check its requests in Verifying Googlebot and other Google crawlers.
- Reverse DNS. Bing and Yandex ask you to run a reverse DNS lookup on the visiting IP and confirm the hostname belongs to them.
A request that claims to be GPTBot but comes from an IP outside OpenAI’s list is not GPTBot.
Which search bots should you allow?
There is no single right answer, but there is a sensible default for most brands that want to be found.
- Allow search crawlers, both classic and AI:
Googlebot,bingbot,OAI-SearchBot,Claude-SearchBot,PerplexityBot,Applebot. Blocking them takes you out of results and AI answers. - Decide on training separately.
GPTBot,ClaudeBot,CCBot,Google-ExtendedandApplebot-Extendedare about model training, not about whether you appear in answers today. - Do not block what you cannot see. A blanket rule such as disallowing every bot with “AI” or “GPT” in its name tends to remove the search bots you wanted to keep.
A robots.txt that keeps AI search open but opts out of training might look like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /This is an example, not a recommendation for every site. Many brands want their content in training data so models know them better. Changes also take time: OpenAI says a robots.txt change can take about a day to apply.
Remember that robots.txt is a request, not a lock. It keeps well-behaved bots out of a page, but it does not hide the page. Private content still needs a login.
How do you see which bots visit your site?
Your server or CDN logs record every request with its user-agent, URL and status code. Filter them by the names above and you will see which bots come, how often, and which pages return errors to them. Our guide to AI crawler traffic analysis walks through that step by step.
Two quick checks catch most problems:
- Is any search bot blocked by mistake? Read your robots.txt with the table above next to it.
- Can bots read the content? If the main text only appears after JavaScript runs, some AI bots will see an empty page. See can AI crawlers read JavaScript websites.
Here is that check in Brantial’s Site Health for webtures.com, the website of the company behind Brantial: every AI search bot is allowed in robots.txt and gets a normal page (HTTP 200) in a live test.
Being crawled is only the first step. Once a bot has your page, an AI answer still has to pick it; that part is explained in what RAG is and how it picks sources.
Frequently asked questions
What is the difference between a search bot and a crawler?
In everyday use there is none; the terms mean the same thing. “Crawler” and “spider” describe how the program moves from link to link, and “search bot” describes what it is for.
Is Googlebot used for AI Overviews?
Yes. Google says AI Overviews and AI Mode rely on the same Googlebot crawling as the rest of Google Search. There is no separate crawler to allow, and no extra technical requirements beyond being indexed and eligible for a snippet.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot. To stay eligible for ChatGPT search, allow OAI-SearchBot; blocking GPTBot only affects possible training use.
What is Google-Extended?
It is a robots.txt token, not a crawler. Disallowing it tells Google not to use your content for training Gemini models and for grounding in Gemini apps and Vertex AI. It does not change how your site appears in Google Search.
Do AI bots follow robots.txt?
Most search and training crawlers do, according to their companies. User-triggered fetchers, which open a page because a user asked, may not. Check each provider’s documentation for the bot in question.