smart_toy
SEO Tools
Robots.txt Generator
Generate a valid robots.txt file to control how search engines crawl and index your website.
public Global Settings
bolt Quick Actions
rule_settings Custom Rules
data_object Live Output
help What is robots.txt?
A `robots.txt` file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests.
Warning: Do not use `robots.txt` to hide private web pages. It is not a security mechanism.
tips_and_updates Best Practices
- check Always include your sitemap.
- check Do not block CSS/JS files. Google needs them to render your page properly.
- check Place the file in the root directory. (e.g. `yourdomain.com/robots.txt`)
shield_lock Privacy Notice
Your robots.txt file is generated entirely inside your browser.
No data, directories, or URLs are ever uploaded to or processed by our servers. All parsing happens via local JavaScript.