Technical SEO

Robots.txt for AI crawlers: the complete guide

Guide · Updated 2026-10-07

AI bots now crawl the web for training and answers. Here's every major one and exactly how to allow or block it.

Quick answer: To allow an AI crawler, add a block like User-agent: GPTBot then Allow: /. To block one, use Disallow: /. Most publishers now allow AI crawlers to be citable in ChatGPT, Perplexity and AI Overviews. Build it in clicks with our robots.txt generator.

Why AI crawlers matter now

AI companies run their own crawlers to gather training data and to fetch pages on demand when answering questions. Your robots.txt is where you decide, per bot, whether they're welcome. Allowing them means your content can be used and cited by AI answer engines — a growing source of visibility. Blocking them keeps your content out of AI, at the cost of that visibility.

The major AI crawlers

User-agentWhoPurpose
GPTBotOpenAITraining data
OAI-SearchBotOpenAIChatGPT search index
ChatGPT-UserOpenAIOn-demand fetch when a user asks
ClaudeBotAnthropicTraining / crawling
Claude-SearchBotAnthropicSearch
PerplexityBotPerplexityIndex for answers
Perplexity-UserPerplexityOn-demand fetch
Google-ExtendedGoogleControls AI (Gemini / AI Overviews) use — separate from Search
CCBotCommon CrawlOpen dataset used by many AI models

How to allow all AI crawlers

This is what most publishers want — full web SEO and AI visibility:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

…and so on for each bot. Our generator writes all of them for you with one toggle each.

How to block AI crawlers

If you'd rather keep your content out of AI training and answers:

User-agent: GPTBot
Disallow: /

Note two things: blocking is only respected by well-behaved bots, and Google-Extended is special — blocking it stops Google using your content for AI features without affecting your normal Search ranking. That makes it the one to use if you want to rank in Google but opt out of its AI.

Where robots.txt goes

At the root of your domain, reachable at yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored. Add a Sitemap: line pointing to your sitemap.xml while you're there.

Important: robots.txt is not deindexing or security

Disallow stops crawling, but a blocked URL can still appear in search if other sites link to it. To remove a page from Google, allow crawling and use a noindex meta tag instead. And robots.txt is a set of polite directives, not a lock — never rely on it to hide sensitive content.

Test before you ship

A single wrong Disallow can hide your whole site. Use the tester in our robots.txt generator to check that a given bot and path resolve to Allowed or Blocked the way you expect.

Frequently asked questions

How do I block GPTBot?

Add this to robots.txt at your site root: 'User-agent: GPTBot' on one line and 'Disallow: /' on the next. Repeat for other AI user-agents you want to block, such as ClaudeBot or PerplexityBot.

Should I allow or block AI crawlers?

Most sites allow them, because being crawled is what lets ChatGPT, Perplexity and Google AI Overviews cite you — a growing traffic source. Block them only if you specifically don't want your content used by AI.

What is Google-Extended?

Google-Extended is a robots.txt token that controls whether Google can use your content for its AI products (Gemini, AI Overviews). Blocking it does not affect your normal Google Search ranking, so it's the way to opt out of Google's AI while staying in Search.

Does Disallow remove my page from Google?

No. Disallow stops crawling but the URL can still be indexed if linked from elsewhere. To deindex a page, allow crawling and add a noindex meta tag instead.

Advertisement

Check your page for SEO & AI search — free

SEOGlint scans any page for on-page SEO, technical health and GEO/AEO readiness in one click.

Run a free audit