AI Crawlers and robots.txt: GPTBot, ClaudeBot, PerplexityBot

In short: AI companies run separate bots for separate jobs, and the difference matters. Some collect content for training models, some build the index behind AI search, and some fetch a page only because a user asked about it. If you block everything with one rule you can disappear from AI answers without meaning to. If you allow everything you may hand over content for training you would rather keep back. This guide lists the main bots from OpenAI, Anthropic, Perplexity and Google, what each is for, and a sensible way to decide.

Key Takeaways

• Training, search and user-initiated fetches are different bots. You can treat them differently in robots.txt.
• OpenAI's documentation states that each setting is independent. GPTBot is for training. OAI-SearchBot surfaces sites in ChatGPT search. ChatGPT-User fetches pages when a user asks.
• Anthropic documents ClaudeBot (training), Claude-SearchBot (search quality) and Claude-User (user requests), and says its bots honour robots.txt.
• Perplexity documents PerplexityBot (search, not used for foundation-model training) and Perplexity-User, which it says generally ignores robots.txt because a user requested the fetch.
• Google uses Googlebot for Search, including its AI features, and a separate token, Google-Extended, to control use for Gemini training and grounding. Google says Google-Extended does not affect inclusion in Search or ranking.
• Changes are not instant. OpenAI and Perplexity both say search settings can take around 24 hours to update.

The Bots at a Glance

Deciding What to Allow

There is no single right answer. It depends on what you want from AI systems.

If you want to be found in AI search, you generally need the search bots to reach your pages: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude and PerplexityBot for Perplexity. Blocking them reduces your chance of being shown in those products.

If you are comfortable with your content being used for training, you can allow GPTBot and ClaudeBot. Many publishers choose to disallow the training bots while allowing the search bots. That is a legitimate choice, and the documentation above describes the bots as separately controllable.

If you want to limit use of your content by Google's Gemini models, use Google-Extended. It does not remove you from Search or change ranking, per Google's documentation. To limit what Search shows from a page, Google points to nosnippet, data-nosnippet, max-snippet and noindex. Blocking Googlebot would block Search.

An Example Policy

This pattern allows search-oriented bots and declines training. It is an illustration, not a recommendation for every site:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Before using anything like this, check your existing robots.txt for broad rules that might override it, and test with each company's current documentation, since tokens and behaviour change.

Things That Catch People Out

• User-initiated fetchers may ignore robots.txt. OpenAI and Perplexity both say their user-triggered bots may not follow it, since a person asked for the page. If content must be private, use authentication, not robots.txt.
• A blanket block hides you. A rule that disallows all unknown bots can shut out search bots you wanted.
• CDN and firewall rules. Security settings can block bots even when robots.txt allows them. Google lists CDN and hosting access among the basics.
• IP blocking is brittle. Anthropic notes that blocking by IP may not reliably deliver an opt-out. It publishes bot IP ranges for allow-listing.
• Subdomains. Rules apply per host, so each subdomain needs its own robots.txt.
• Timing. Allow time after changes, and do not conclude it failed after an hour.

The Short Version

Decide separately about training, AI search and user-triggered fetching. Allow the search bots if you want to be visible in AI answers, decide on the training bots based on your own policy, and use real access controls for anything private. Review the policy every few months, because the list of bots keeps growing.

This is one part of making your site visible to AI. For the rest, read what GEO is and our answer engine optimization service.

Related Insights

• What Is GEO? Generative Engine Optimization vs AEO vs SEO
• Where AI Answers Get Their Sources, and Why Listings Matter as Much as Your Site
• Google AI Overviews Antitrust Ruling: What It Means for AI Search and GEO

Sources

• OpenAI, Overview of OpenAI crawlers, read November 3, 2026
• Anthropic, Does Anthropic crawl data from the web, and how can site owners prevent crawling?, article dated April 7, 2026
• Perplexity, Perplexity crawlers, read November 3, 2026
• Google, List of Google's common crawlers, read November 3, 2026
• Google Search Central, AI features and your website, read November 3, 2026