Robots.txt Generator

Create a valid robots.txt: allow everything, block everything, or fine-tune disallowed paths per crawler — with your sitemap attached.

robots.txt
—
Advertisement

How to use Robots.txt Generator

  1. Pick a preset or choose custom.
  2. List the paths to disallow (one per line).
  3. Add your sitemap URL and copy the file to your site root.

robots.txt is a request, not a lock — compliant crawlers obey it, malicious ones ignore it. For true privacy you need authentication or noindex headers.

More tools on Webmaster Tools

FAQ

Allow-all vs empty file?

Same effect for compliant crawlers; an explicit file is still better because it also carries your Sitemap directive.

Does Disallow hide pages from Google?

No — blocked pages can still appear by URL without a snippet. Use noindex meta tags to keep them out of results entirely.

Is crawl-delay respected?

By many crawlers (Bing, Yandex) but not Google, which manages its own crawl rate.

Like this tool?

Get Very Useful Tools Pro — one $6/month subscription for all tools: ad-free once ads launch, plus bigger batches and higher limits.

Go Pro