Robots.txt Generator
Create a robots.txt file to control search engine crawling.
Robots.txt is a text file at the root of your website that tells search engine crawlers which pages to crawl and which to ignore. A well-configured robots.txt protects staging environments, admin pages, and private URLs from indexation while ensuring your important content is fully crawlable. This generator creates a properly formatted robots.txt based on your website type (WordPress, custom, e-commerce) with the most common directives.
📋 How to Use This Calculator
Select your CMS or website type (WordPress, Shopify, custom). The generator pre-fills common directories to disallow (wp-admin, cart, checkout, staging). Add custom allow and disallow rules using the rule builder. Optionally add your sitemap URL and crawl delay. The generator shows a preview and explains what each rule does. Paste the output into a text file named "robots.txt" and upload to your domain root.
💡 Key Facts & Information
Robots.txt syntax: User-agent: * (applies to all bots), User-agent: Googlebot (Google only). Disallow: /admin/ (block path). Allow: /admin/public/ (allow subset of blocked path). Crawl-delay: 10 (seconds between requests — not respected by Googlebot; use Google Search Console crawl rate settings instead). Sitemap: https://example.com/sitemap.xml. Common rules: WordPress: disallow /wp-admin/, /wp-includes/, /?s= (search results). E-commerce: disallow /cart, /checkout, /account. Always disallow staging/test environments. Important: robots.txt is publicly visible — do not mention sensitive URLs even to disallow them. Use meta noindex for pages you want hidden from index but not from crawlers.