Robots.txt Generator

Robots.txt is a plain text file at the root of a domain that tells crawlers which parts of the site they should not request. The tool builds one from rules you pick, with a preview of the finished file and a check for the mistakes that make a robots.txt do the opposite of what its author intended.

Those mistakes are specific and common. Disallow with a single slash blocks the entire site, and one stray character is all it takes. Rules are matched by prefix, so disallowing /admin also blocks /administrator-notes. Blank lines inside a group end that group. And the file is only read at the root of the domain, so putting it in a subfolder achieves nothing.

The most important thing to understand is what robots.txt does not do. It stops well-behaved crawlers requesting a page; it does not stop that page appearing in search results if other sites link to it, and it does not keep anything private. A blocked page can still be indexed with no description. To keep a page out of an index, allow the crawl and use a noindex tag; to keep it private, use a login.

• Per-crawler groups with allow and disallow rules
• Presets for an open site, WordPress and staging
• Sitemap references, including several
• Optional crawl-delay with a note on which crawlers honour it
• Warns when a rule would block the whole site
• Explains that blocking is not the same as removing from an index
• Copy or download the finished file

The robots.txt generator is used by writers, developers, students, marketers and anyone else who needs the job done once without installing software. Common cases include:

• Setting up robots.txt for a new site
• Keeping crawlers out of admin and search result pages
• Blocking a staging site from being crawled
• Adding a sitemap reference so crawlers find your URLs
• Allowing one crawler while restricting others