Robots.txt Generator
Create a valid robots.txt: allow everything, block everything, or fine-tune disallowed paths per crawler — with your sitemap attached.
robots.txt
—
Advertisement
How to use Robots.txt Generator
- Pick a preset or choose custom.
- List the paths to disallow (one per line).
- Add your sitemap URL and copy the file to your site root.
robots.txt is a request, not a lock — compliant crawlers obey it, malicious ones ignore it. For true privacy you need authentication or noindex headers.
More tools on Webmaster Tools
FAQ
Allow-all vs empty file?
Same effect for compliant crawlers; an explicit file is still better because it also carries your Sitemap directive.
Does Disallow hide pages from Google?
No — blocked pages can still appear by URL without a snippet. Use noindex meta tags to keep them out of results entirely.
Is crawl-delay respected?
By many crawlers (Bing, Yandex) but not Google, which manages its own crawl rate.