Free Robots.txt Generator
Generate a properly formatted robots.txt file. Control which pages search engine bots can crawl and index. Free, with no sign-up required — everything happens right in your browser, so nothing you enter is sent to a server.
🔧 Robots.txt Generator
How the Robots.txt Generator Works
Add one or more rules, choosing which crawler they apply to (all bots, Googlebot or Bingbot), whether to Allow or Disallow, and the path, such as /admin/. Optionally add the address of your sitemap. The tool writes a correctly formatted robots.txt, with a User-agent line for each crawler followed by its rules, and a Sitemap line at the end. Copy it or download it.
Upload the file to the root of your domain so it is reachable at yourdomain.com/robots.txt. Robots.txt tells well-behaved crawlers which paths not to fetch, but it is not a security feature, since the file is public and bad bots ignore it, and a blocked page can still appear in search results if other sites link to it. To keep a page out of search results, use a noindex tag instead, and never block CSS or JavaScript files that search engines need to render your pages.
Practical Use Cases
- Stop search engines from crawling admin, cart or internal search pages.
- Point crawlers to your XML sitemap.
- Create a starter robots.txt for a new website.
- Block a specific crawler from one section while allowing others.
Tips for Best Results
- Test your rules in Google Search Console's robots.txt report after you upload the file, since a typo can block a whole site.
- Never block CSS, JavaScript or image folders that your pages need, because search engines use them to render and understand the page.
- Add your sitemap line, and use noindex, not Disallow, for pages that you want kept out of search results.