Quick answer: To allow an AI crawler, add a block like User-agent: GPTBot then Allow: /. To block one, use Disallow: /. Most publishers now allow AI crawlers to be citable in ChatGPT, Perplexity and AI Overviews. Build it in clicks with our robots.txt generator.
Why AI crawlers matter now
AI companies run their own crawlers to gather training data and to fetch pages on demand when answering questions. Your robots.txt is where you decide, per bot, whether they're welcome. Allowing them means your content can be used and cited by AI answer engines — a growing source of visibility. Blocking them keeps your content out of AI, at the cost of that visibility.
The major AI crawlers
| User-agent | Who | Purpose |
|---|---|---|
GPTBot | OpenAI | Training data |
OAI-SearchBot | OpenAI | ChatGPT search index |
ChatGPT-User | OpenAI | On-demand fetch when a user asks |
ClaudeBot | Anthropic | Training / crawling |
Claude-SearchBot | Anthropic | Search |
PerplexityBot | Perplexity | Index for answers |
Perplexity-User | Perplexity | On-demand fetch |
Google-Extended | Controls AI (Gemini / AI Overviews) use — separate from Search | |
CCBot | Common Crawl | Open dataset used by many AI models |
How to allow all AI crawlers
This is what most publishers want — full web SEO and AI visibility:
User-agent: GPTBotAllow: /User-agent: ClaudeBotAllow: /
…and so on for each bot. Our generator writes all of them for you with one toggle each.
How to block AI crawlers
If you'd rather keep your content out of AI training and answers:
User-agent: GPTBotDisallow: /
Note two things: blocking is only respected by well-behaved bots, and Google-Extended is special — blocking it stops Google using your content for AI features without affecting your normal Search ranking. That makes it the one to use if you want to rank in Google but opt out of its AI.
Where robots.txt goes
At the root of your domain, reachable at yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored. Add a Sitemap: line pointing to your sitemap.xml while you're there.
Important: robots.txt is not deindexing or security
Disallow stops crawling, but a blocked URL can still appear in search if other sites link to it. To remove a page from Google, allow crawling and use a noindex meta tag instead. And robots.txt is a set of polite directives, not a lock — never rely on it to hide sensitive content.
Test before you ship
A single wrong Disallow can hide your whole site. Use the tester in our robots.txt generator to check that a given bot and path resolve to Allowed or Blocked the way you expect.
Frequently asked questions
How do I block GPTBot?
Add this to robots.txt at your site root: 'User-agent: GPTBot' on one line and 'Disallow: /' on the next. Repeat for other AI user-agents you want to block, such as ClaudeBot or PerplexityBot.
Should I allow or block AI crawlers?
Most sites allow them, because being crawled is what lets ChatGPT, Perplexity and Google AI Overviews cite you — a growing traffic source. Block them only if you specifically don't want your content used by AI.
What is Google-Extended?
Google-Extended is a robots.txt token that controls whether Google can use your content for its AI products (Gemini, AI Overviews). Blocking it does not affect your normal Google Search ranking, so it's the way to opt out of Google's AI while staying in Search.
Does Disallow remove my page from Google?
No. Disallow stops crawling but the URL can still be indexed if linked from elsewhere. To deindex a page, allow crawling and add a noindex meta tag instead.