At some point in the last two years, AI companies started reading your website. GPTBot, ClaudeBot, PerplexityBot, plus a dozen lesser-known names: they all come knocking, and many robots.txt files now carry a hastily copied block list against them. The problem: those lists throw everything into one pot. And whoever blocks the wrong bot disappears from ChatGPT answers without noticing.
Deciding which AI crawlers to allow is not a technical footnote; it is a visibility decision. This piece sorts the bots into three classes, walks through the major vendors, and ends with ready-made robots.txt blocks for the two positions a site can reasonably take.
Three kinds of bots, three different decisions
Nearly every AI vendor runs several crawlers with different jobs, and the consequence of blocking each one is different.
Training crawlers collect text used to train future models. Block them and your content is not in upcoming model versions. Your visibility in AI answers with web search is barely affected, because those lean on current search results, not training knowledge. This is the only one of the three blocks you can defend without sacrificing visibility.
Search crawlers fill the search index from which AI answers pull their sources. Block them and the system can no longer find and cite your pages. This is the block that hurts: it destroys exactly the visibility described in Getting found in ChatGPT.
User fetches happen when a human gives the system a concrete task: "summarize this page", "check the opening hours on this site". The bot then fetches a single URL on a person's behalf. Block it and that person gets an error message instead of your content. Nothing is won.
The rule of thumb: blocking training is a matter of taste, blocking search costs visibility, blocking user fetches only annoys humans.
Who comes knocking
The names that matter, sorted by vendor and class:
OpenAI runs three: GPTBot (training), OAI-SearchBot (the search index behind ChatGPT answers) and ChatGPT-User (user fetches). The widely copied block lists from 2023 block GPTBot. If you pasted one, check whether OAI-SearchBot slipped into the list later.
Anthropic mirrors the pattern: ClaudeBot (training), Claude-SearchBot (search index) and Claude-User (user fetches).
Perplexity uses PerplexityBot for its own index and Perplexity-User for fetches on a user's behalf. Perplexity is a pure answer engine: blocking PerplexityBot means not existing there at all.
Google is the special case that gets misunderstood most often. The AI Overviews in search are fed from the normal Googlebot index. There is no separate "AI Overviews bot" you could block without leaving Google search altogether. Google-Extended is not a crawler but a control token in robots.txt: it only governs whether your content may be used to train the Gemini models. It has no effect on AI Overviews.
The rest: Applebot-Extended (Apple, training opt-out, analogous to Google-Extended), CCBot (Common Crawl, an open web archive many models were trained on), Meta-ExternalAgent (Meta, training) and Bytespider (ByteDance). The last one is notorious for simply ignoring robots.txt.
Which position to take
For the vast majority of business websites the math is simple: your content is advertising for your service, not the service itself. A trade business, a SaaS, a law firm lives on being found and recommended, including by a language model that knows from training that the company exists. Then the right position is: let everything through, training crawlers included. Every block can only cost visibility and earns nothing.
The case is different when your content is your product, say as a publisher, a paid trade portal or a photographer. Then the training block is a legitimate business decision: you do not want a model reproducing your product for free. But even then: let search crawlers and user fetches through, or you vanish from the very answers that could bring you customers.
What practically nobody needs is the total block, "everything with AI in its name, out". It is the default setting of 2023's outrage, and today it measurably costs reach.
The two robots.txt blocks
Position one, maximize visibility: nothing to do here, because without rules all bots may read everything anyway. Just check that no old blocks are left in the file that need to go.
Position two, training out, visibility in:
# No training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
# Search and user fetches stay open: OAI-SearchBot, Claude-SearchBot,
# PerplexityBot, ChatGPT-User etc. get no rules and may read everything.
The file belongs in the root directory, right next to your llms.txt if you have one. The two complement each other: robots.txt says who should not read, llms.txt tells those who may read where the important things are.
What robots.txt can do, and the trap nobody sees
Two caveats belong here. First, robots.txt is a request, not a lock: serious vendors respect it, Bytespider and assorted anonymous scrapers do not. If you really want to block, block at server level, via firewall or CDN rule, using the vendors' published IP ranges.
Second, and this is the trap: you may be blocking already without knowing it. Large CDN and hosting providers have built in switches that turn AI crawlers away wholesale. At Cloudflare, blocking AI crawlers has even been the default for newly created domains since mid-2025. These blocks fire before your robots.txt: the bot gets an error no matter what your file says. If your site never shows up in AI answers although copy and rankings are fine, an activated bot shield at your host is suspect number one.
Checking who actually visits
Whether and how often the AI crawlers visit is in your server logs: search for the user-agent names from this piece. Two things can be read from them. If OAI-SearchBot or PerplexityBot drops by regularly, you are in their index. Good. If nothing comes at all although nothing is blocked, classic findability is usually what is missing; then the groundwork from Google Search Console comes first, before you tune any AI details.
A word of caution in the other direction: user agents can be forged. If an alleged GPTBot floods your site with requests, check the IP against the vendor's published ranges before blaming OpenAI. Usually it is a scraper in disguise.
Frequently asked questions
Should I block GPTBot?
Only if your content itself is your product, say as a publisher or paid portal, and you do not want models trained on it. For ordinary business websites the block earns nothing and gives away the chance that future models know your company from training. Visibility in ChatGPT answers with web search depends on OAI-SearchBot anyway, not on GPTBot.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects training material for future OpenAI models. OAI-SearchBot fills the search index from which ChatGPT pulls its sources when answering with web search. If you want to be cited in ChatGPT answers, OAI-SearchBot must be allowed through; GPTBot is irrelevant for that.
Does Google-Extended block AI Overviews?
No. Google-Extended only governs whether your content may be used to train the Gemini models. AI Overviews in Google search are built from the normal search index; the only ways out are restricting snippets or leaving the index entirely, with all the consequences for classic visibility.
Do AI crawlers respect robots.txt?
The major vendors like OpenAI, Anthropic, Google and Apple document their crawlers and follow the rules. Some others, Bytespider first among them, ignore them regularly. Reliable blocking only works at server level, based on published IP ranges.
Why does my site never appear in AI answers although I block nothing?
Most common cause: a bot shield at your CDN or host turns the crawlers away before your robots.txt is ever read. At some providers this is now the default. Second most common: the page is too hard to find in classic search for its topic, because AI answers pick their sources from the top search results.