Robots.txt Generator
Generate a custom, valid robots.txt file to guide search engine crawlers.
1. Basic Configuration
2. Search Engine Bot Settings
Googlebot
Googlebot-Image
Bingbot
Baiduspider
Yandex
DuckDuckBot
3. Custom Paths / Directives
User-agent: * → Disallow: /admin/
User-agent: * → Disallow: /api/
Add New Crawler Rule
Robots.txt Output Preview
Frequently Asked Questions (FAQ) – Robots.txt
What is a robots.txt file?
A robots.txt file is a simple text file placed in your website's root directory that tells search engine crawlers (like Googlebot) which pages or sections they are allowed or not allowed to visit and index.
Where should the robots.txt file be uploaded?
It must be uploaded directly to the root directory of your domain (e.g., https://example.com/robots.txt).
How does this generator handle crawler instructions?
The tool lets you set a default policy for all crawlers, define custom crawl delays, declare your Sitemap URL, and customize rules for specific bots such as Googlebot, Bingbot, and Baidu.