robots.txt Generator
Builds explicit allow or block groups for documented crawler tokens and a sitemap declaration.
Generated file
User-agent: * Allow: /
What this tool checks
Builds explicit allow or block groups for documented crawler tokens and a sitemap declaration.
How it works
Choose rules deliberately, generate the file, review every group and publish it at the origin root as /robots.txt.
Common mistakes
Do not use robots.txt for private data, authentication or guaranteed enforcement. Test the public response after CDN and CMS deployment.