Vibeking

robots.txt and AI crawlers: which bots should you allow?

By Vibeking · Published 30 September 2026 · Facts checked on 30 September 2026

Don't block all AI bots in one go. OpenAI, Anthropic and other AI companies use separate bots for training models and for search.

So you can refuse training and still stay visible in answers from ChatGPT, Claude and Perplexity. The condition is that you allow their search bots.

Three kinds of AI bots

  • Training bots collect pages to train AI models. OpenAI explicitly says that this choice is separate from its search bot, and Anthropic uses separate bots for training and search.
  • Search bots build the index from which an assistant draws its sources. If you block them, you are shown as a source in their answers much less often, or not at all.
  • Bots that fetch on request open a page because a user asks for it. OpenAI and Meta say that robots.txt may not apply to them. Google and Perplexity say that such bots generally ignore robots.txt. Anthropic says that all its bots follow robots.txt.

Which bot does what

BotFromUsed forWhat blocking does
GPTBotOpenAIModel trainingYou indicate that your content may not be used to train OpenAI's foundation models
OAI-SearchBotOpenAISearch in ChatGPTYou no longer appear in ChatGPT's search answers, at most as a navigation link; takes effect after about 24 hours
ChatGPT-UserOpenAIFetching on requestrobots.txt may not apply
ClaudeBotAnthropicModel trainingYou indicate that new content may not be used for training
Claude-SearchBotAnthropicClaude's search indexYour content is not indexed for the search function
Claude-UserAnthropicFetching on requestClaude does not fetch your page when a user asks for it
PerplexityBotPerplexitySearch results, not for trainingYou may no longer appear in Perplexity's search results
Perplexity-UserPerplexityFetching on requestGenerally ignores robots.txt
Google-ExtendedGoogleNot a separate bot but a token: training of Gemini and sources for the Gemini appsNo effect on Google Search or your ranking, but it does affect use in the Gemini apps
Applebot-ExtendedAppleToken for training Apple modelsYour pages stay in Apple's search results
CCBotCommon CrawlOpen web archive, a well-known source for AI trainingYour pages no longer end up in new archives
meta-externalagentMetaTraining and indexing for Meta productsYour content is no longer fetched for these purposes
meta-webindexerMetaMeta AI search resultsLess visible in Meta AI

Googlebot, Bingbot and Applebot are the regular search bots: always allow them. To appear as a source link in AI Overviews or AI Mode, a page must be indexed in Google Search and eligible to be shown with a snippet. Since 31 August 2026, you can use Search Console, under Settings > Search generative AI, to exclude your site from AI Overviews, AI Mode and the generative AI features in Discover without blocking Googlebot. That setting changes nothing about training; for that, you still need Google-Extended.

Microsoft has no separate AI token for robots.txt. For Bing's chat answers, Microsoft pointed to the NOCACHE and NOARCHIVE meta tags in 2023.

Refuse training, stay visible: an example

# No training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

# Allow AI search services and fetching on request
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: meta-webindexer
Allow: /

# All other bots, such as Googlebot, Bingbot and Applebot
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Three things to watch out for:

  1. A bot with its own group ignores the User-agent: * group. If that group contains a rule for a protected directory, such as Disallow: /account/, repeat it in every group.
  2. Blocking Google-Extended also stops your content from being used as a source in the Gemini apps. If you do want to be mentioned there, leave that line out.
  3. robots.txt applies per domain and per subdomain. OpenAI, Perplexity and Meta process a change within about 24 hours.

Watch out if you use Cloudflare

On 1 July 2026, Cloudflare announced that customers who block training there would, from 15 September, also block multi-purpose bots such as Googlebot, Applebot and Bingbot. On 15 September 2026, the Disallow AI Training setting was added. It puts a ban on training in your robots.txt, blocks training bots such as GPTBot and ClaudeBot, and allows Googlebot, Bingbot and Applebot for search. Cloudflare has converted existing training blocks to Disallow AI Training. Bing does not yet read that ban in robots.txt; according to Cloudflare, Microsoft aims to support it in early 2027.

If you choose Block under Training in Cloudflare, Googlebot, Bingbot and Applebot are also blocked for search and Google can no longer fetch your site. Block on pages with ads does the same on pages with advertisements. So always check that setting alongside your robots.txt.

On 24 September 2025, Cloudflare also introduced Content Signals: a line such as Content-Signal: search=yes, ai-input=yes, ai-train=no, which lets you indicate what your content may be used for. Cloudflare itself does not call it a technical block, and we found no AI company that has publicly committed to following the signals.

Is robots.txt legally binding?

Technically, robots.txt is an agreement, not a lock: the standard (RFC 9309) says that the rules are not a form of access authorisation. Legally, however, it can count.

Under article 4 of the European copyright directive (the DSM directive) and article 15o of the Dutch Copyright Act (Auteurswet), text and data mining is not permitted if the rightholder has expressly reserved it in an appropriate manner. For online content, this refers to machine-readable means. Since 2 August 2025, providers of general-purpose AI models must have a policy to identify and comply with such a reservation (AI Act, article 53). For models already on the market before that date, this applies by 2 August 2027 at the latest.

How courts weigh a rule in robots.txt is still in flux. In 2024, the Amsterdam District Court ruled in a case brought by DPG Media and Mediahuis against HowardsHome that a valid reservation had not been demonstrated, and the German Federal Court of Justice (Bundesgerichtshof) is expected to rule in a similar case on 17 December 2026. This article is not legal advice.

How to check it

Open yourdomain.com/robots.txt and look for the names from the table. If OAI-SearchBot, Claude-SearchBot or PerplexityBot falls under a Disallow: /, or if one of those bots has no group of its own while User-agent: * has a Disallow: /, you are excluding yourself from those AI search services. A block in Cloudflare or a firewall does not show up in robots.txt; check those separately.

The free Vibeking SEO check reads your robots.txt and reports whether you block OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot or Google-Extended. Google-Extended is included because that token also determines whether the Gemini apps use your content as a source. If you deliberately block it to refuse training, as in the example above, that finding is to be expected. The check shows training bots such as GPTBot, ClaudeBot, CCBot, Applebot-Extended and meta-externalagent for information only.

Sources

How is your site doing?

The free Vibeking SEO check runs 30 checks and 7 AI checks on your site and turns every finding into a fix task with guidance.

Run the free SEO check

More in the knowledge base

How does your website score?

Run the free check: 30 checks and 7 AI checks, with a fix task for every finding. No payment details needed.