What is a robots.txt file?
robots.txt is a plain-text file at the root of your website โ reachable at yourdomain.com/robots.txt โ that tells crawlers which parts of your site they may or may not request. It's the first file most bots look at. Well-behaved crawlers (Googlebot, Bingbot, and the major AI crawlers) respect it; it is a set of directives, not a security wall, so it won't stop a determined bad actor.
Controlling AI crawlers (the new part)
AI companies now run their own crawlers to gather training data and to power AI search answers. You can decide, per bot, whether they're allowed. The main ones:
- GPTBot โ OpenAI's training crawler.
- OAI-SearchBot / ChatGPT-User โ OpenAI's search and on-demand fetchers.
- ClaudeBot / Claude-SearchBot โ Anthropic's crawlers.
- PerplexityBot / Perplexity-User โ Perplexity's crawlers.
- Google-Extended โ controls use of your content for Google's AI (Gemini / AI Overviews) without affecting normal Search.
- CCBot โ Common Crawl, used by many AI datasets.
If you want AI visibility (as most publishers now do), allow them โ being citable in ChatGPT, Perplexity and AI Overviews is a growing traffic source. If you'd rather keep your content out of AI, block them. This tool lets you choose each one.
How Disallow and Allow work
Each block starts with a User-agent: line, then one or more Disallow: or Allow: paths. Disallow: / blocks everything; Disallow: (empty) blocks nothing. When rules conflict, most modern crawlers follow the longest (most specific) matching path. Our tester uses the same longest-match logic so you can check a path before you ship.
Important: robots.txt does not deindex
Blocking a URL with Disallow stops crawling, but if other sites link to that URL it can still show up in Google without a description. To actually remove a page from the index, allow crawling and add a <meta name="robots" content="noindex"> tag to the page instead.
Frequently asked questions
Where does robots.txt go?
At the root of your domain, reachable at yourdomain.com/robots.txt. A robots.txt inside a subfolder is ignored.
Should I block AI crawlers?
Only if you don't want your content used by or cited in AI. Most sites now allow them for the AI-search visibility.
Does Disallow remove a page from Google?
No โ it stops crawling but the URL can still be indexed if linked. Use a noindex meta tag to deindex.
Is this tool free?
Yes โ generating, testing, copying and downloading is completely free and runs entirely in your browser.