🛠 Free SEO Tool
Robots.txt Generator
Build a professional robots.txt file for your website. Control which bots can crawl your pages and protect sensitive directories.
Global Settings
Used to generate your sitemap line automatically.
Leave blank to auto-generate from site URL above.
Googlebot Presets
Allow Googlebot
Let Google crawl your full site
Block Google Images Bot
Prevent image indexing (rare use case)
Block Adsbot-Google
Block Google's ad-scanning bot
Other Search Engines
Allow Bingbot
Microsoft Bing crawler
Allow Yandex
Russian search engine bot
Allow DuckDuckBot
DuckDuckGo crawler
AI & Scraper Bots
Block GPTBot (OpenAI)
Prevent ChatGPT from training on your content
Block CCBot (Common Crawl)
Used to train many AI models
Block Semrush Bot
SEMrush scraper bot
Block AhrefsBot
Ahrefs backlink crawler
Advanced Rules
One path per line. E.g: /admin/ /private/ /wp-login.php
Force-allow specific paths even if parent is disallowed.
Slow down crawlers. 0 = none.
Which bot group.
Custom Bot Rules
Blogger-Specific Paths
Block /search pages
Prevents duplicate content issues on Blogger
Block /feeds/ directory
Reduces duplicate crawling of RSS feeds
Block label pages
Prevents /search/label/ pages from crawling
💡 Pro Tip for Blogger Users
After generating, go to Blogger → Settings → Crawlers and indexing, enable Custom robots.txt, and paste your output there.
✅ Your robots.txt Output
