Skip to content
Try it free
Get Search Signal

Robots.txt Generator

Create a robots.txt file to instruct search engine bots on how to crawl your website.

About the Robots.txt Generator

The Robots.txt Generator builds correct robots.txt rules for your site — allow and disallow directives, crawl-delay where supported, and sitemap references — with presets for AI crawler access.

Robots.txt is now a strategic file: beyond classic crawl control, it governs whether GPTBot, ClaudeBot, PerplexityBot, and Google-Extended may fetch your pages for AI answers. Block them and you opt out of citations; allow them deliberately with a documented rationale.

Generate the file, upload it to your domain root, and test it before relying on it. Free with no login.

Frequently asked questions

What is robots.txt?▾
Robots.txt is a plain-text file at your domain root that tells crawlers which paths they may fetch. It is advisory — compliant bots obey it — and it is the first file any serious crawler requests.
Should I block AI crawlers in robots.txt?▾
Only with a deliberate strategy. Blocking AI bots opts you out of training and retrieval, which also opts you out of AI citations. Most sites seeking visibility should explicitly allow the major AI crawlers.
Which AI crawlers should I allow?▾
At minimum GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, and CCBot (Common Crawl). List each explicitly so future policy changes are intentional.
How do I test my robots.txt?▾
Fetch /robots.txt and confirm it returns 200 with your intended rules, then use a robots.txt tester to check specific bot-and-URL combinations, including your AI crawler allows.

Related tools

AEO Analyzer

Score your content for answer engines — ChatGPT, Perplexity and AI Overviews. Get extractable-answer and citation-readiness fixes. Pro.

GEO Optimizer

Optimize content so generative AI engines cite your brand. Get entity, schema and extractability recommendations. Pro plan feature.

Schema Generator

Generate clean JSON-LD structured data for Organization, Article, FAQPage, Product and more. Copy-paste ready schema markup, no signup needed.

Backlink Checker

Analyze external backlinks pointing to any URL or domain. See link quality, anchors and referring domains. Sign in for full data.