Robots.txt & Sitemap Generator
Generate a robots.txt file with per-user-agent allow and disallow rules and a sitemap reference, then copy or download the output ready for your root directory.
Rule #1
robots.txt output
User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml
Upload the downloaded file to your site root so it is served at /robots.txt.
About the Robots Generator
The robots.txt file tells search engine crawlers which parts of your site they may crawl. A well-formed robots.txt prevents indexing of thin or private pages, manages crawl budget, and points crawlers to your sitemap. This generator builds the file for you and lets you download it ready for your site root.
How to use the Robots.txt Generator
- 1Add user-agent rules using the form. Pick a common user-agent from the dropdown or type a custom one, then add the paths you want to allow or disallow.
- 2Enter your sitemap URL in the Sitemap field. The generator includes it as a Sitemap directive at the end of the file.
- 3Review the generated robots.txt in the output panel. It updates live as you add or remove rules.
- 4Click Copy to copy the text, or Download to save it as robots.txt and upload it to your site root.
Best practices
- Never use robots.txt to hide sensitive pages. Blocked pages can still be indexed if they are linked from elsewhere. Use a noindex meta tag or authentication instead.
- Reference your XML sitemap so crawlers discover all your important URLs efficiently.
- Be specific with disallow paths. A broad Disallow: / blocks the entire site from a crawler.
- Test your robots.txt with Google Search Console after deploying to confirm it behaves as expected.
- Keep the file minimal. Most sites only need to block a few paths like /admin/ or /cart/.
Managing crawl budget on a large site?
Enterprise SEO tools visualize crawl paths and surface wasted crawl budget so you can direct Google to your most valuable pages.