Tool 07 · Technical
Robots.txt Generator
Create a correct robots.txt in a minute. Start from a preset, add the paths to keep crawlers out of, decide on AI crawlers, and add your sitemap.
robots.txt
Save as robots.txt in the root of your domain, e.g. example.com/robots.txt.
Before you block anything
robots.txt stops crawling, not indexing. If a blocked URL is linked from elsewhere, Google can still index it without seeing the content. To remove a page from search, leave it crawlable and add a noindex tag instead. Never block CSS or JavaScript files that pages need to render, and never ship a staging robots.txt that blocks everything to your live site.
About AI crawlers
Blocking AI crawlers is a business decision, not a technical one. Training crawlers such as GPTBot, ClaudeBot or Google-Extended collect content for model training; some assistants also fetch pages live to answer questions and cite sources. Blocking reduces how your content is used, and may also reduce how often you are cited in AI answers. The AI SEO guide covers the trade-off. Well-behaved crawlers follow robots.txt; it is not a security control.
Work with me
Need help applying this?
Reading is the easy part. If you want this implemented on your site, tell me what you're working on.
Work With Me Follow the SEO Roadmap