Robots.txt Generator
Generate, validate, and fetch real robots.txt files from any website
🔍 Real data fetcher
⚡ Live preview
🤖 SEO optimized
📊 W3C compliant
🔄 Fetch existing
Robots.txt Builder
Build your robots.txt with real-time preview
Fetch existing robots.txt from any website
📄 Fetched robots.txt
Sitemap URLs (optional)
Crawl Delay (seconds)
seconds between crawl requests (for all bots)
Default
Allow all bots
WordPress
Optimized for WP
E-commerce
WooCommerce, Shopify
Blog
Blogger, Medium style
Forums
phpBB, vBulletin
Portfolio
Creative sites
News
News portals
Educational
LMS, courses
Click any preset to load optimized robots.txt rules
Live Preview & Analysis
Real-time robots.txt preview with SEO insights
📄 robots.txt
Live
# Your robots.txt will appear here
User-agent: *
Allow: /
# Sitemaps
# Sitemap: https://example.com/sitemap.xml
📊
Real-time SEO Analysis
User-agent: * - All search bots allowed
Add Sitemap directive for better indexing
No Disallow rules defined
Test if a URL is allowed
Robots.txt Guide
1 What is robots.txt?
A file that tells search engines which pages to crawl and index. It's the first file search engines look for when crawling a website.
2 User-agent directive
Specifies which search engine bot the rules apply to. Use * for all bots, or specific names like Googlebot, Bingbot, etc.
3 Allow vs Disallow
Disallow blocks access to paths, Allow explicitly permits access. Order matters - most specific rules take precedence.