๐Ÿค– Crawlers & AI bots

Free Robots.txt Generator & Tester

Build a valid robots.txt in clicks โ€” set crawl rules, allow or block AI crawlers like GPTBot and ClaudeBot, add your sitemap, then test any URL path. Free, no sign-up.

AI crawlers โ€” allow or block each
Custom rules (optional)
Advertisement

๐Ÿงช Test a URL against rules

What is a robots.txt file?

robots.txt is a plain-text file at the root of your website โ€” reachable at yourdomain.com/robots.txt โ€” that tells crawlers which parts of your site they may or may not request. It's the first file most bots look at. Well-behaved crawlers (Googlebot, Bingbot, and the major AI crawlers) respect it; it is a set of directives, not a security wall, so it won't stop a determined bad actor.

Controlling AI crawlers (the new part)

AI companies now run their own crawlers to gather training data and to power AI search answers. You can decide, per bot, whether they're allowed. The main ones:

If you want AI visibility (as most publishers now do), allow them โ€” being citable in ChatGPT, Perplexity and AI Overviews is a growing traffic source. If you'd rather keep your content out of AI, block them. This tool lets you choose each one.

How Disallow and Allow work

Each block starts with a User-agent: line, then one or more Disallow: or Allow: paths. Disallow: / blocks everything; Disallow: (empty) blocks nothing. When rules conflict, most modern crawlers follow the longest (most specific) matching path. Our tester uses the same longest-match logic so you can check a path before you ship.

Important: robots.txt does not deindex

Blocking a URL with Disallow stops crawling, but if other sites link to that URL it can still show up in Google without a description. To actually remove a page from the index, allow crawling and add a <meta name="robots" content="noindex"> tag to the page instead.

Frequently asked questions

Where does robots.txt go?

At the root of your domain, reachable at yourdomain.com/robots.txt. A robots.txt inside a subfolder is ignored.

Should I block AI crawlers?

Only if you don't want your content used by or cited in AI. Most sites now allow them for the AI-search visibility.

Does Disallow remove a page from Google?

No โ€” it stops crawling but the URL can still be indexed if linked. Use a noindex meta tag to deindex.

Is this tool free?

Yes โ€” generating, testing, copying and downloading is completely free and runs entirely in your browser.

Not sure if your robots.txt helps or hurts AI visibility?

SEOGlint audits any page for SEO, technical health and GEO/AEO โ€” including AI-crawler access and robots rules.

Run a free audit