Home› Generators› Robots.txt Generator
Robots.txt Generator
Compose crawl rules per bot without syntax mistakes.
About the Robots.txt Generator
Compose crawl rules without the syntax mistakes that make a robots.txt file do the opposite of what you intended. The generator writes the per-bot blocks, the allow and disallow rules, the crawl delay and the sitemap line in the right order.
The file is far more dangerous than it looks. A single Disallow: / left over from a staging server will empty a site out of search results within days, and because the file is small and rarely opened, nobody notices until traffic collapses.
Two rules to keep in mind. Never block your CSS or JavaScript: Google renders pages like a browser and blocking the assets makes it see a broken layout. And remember that blocking a URL does not remove it from the index — it just means Google cannot look at it, so it may keep showing the bare URL indefinitely.
Common questions
Can one wrong line really take my site out of search results?
Yes. A leftover Disallow: / from a staging setup blocks the entire site from crawling, and because the file is rarely opened, it can go unnoticed for weeks while traffic falls.
Should I block my CSS and JavaScript files?
No. Google renders pages like a browser, and blocking those assets makes it see a broken, unstyled layout — which can cost mobile-usability credit.
Does Crawl-delay work on Google?
No — Googlebot ignores it entirely. Bing and Yandex do honour it, so it's only useful if those engines are crawling too aggressively.