Skip to content

Free Robots.txt Generator

Generate a properly formatted robots.txt file. Control which pages search engine bots can crawl and index. Free, with no sign-up required — everything happens right in your browser, so nothing you enter is sent to a server.

🔧 Robots.txt Generator

How the Robots.txt Generator Works

Add one or more rules, choosing which crawler they apply to (all bots, Googlebot or Bingbot), whether to Allow or Disallow, and the path, such as /admin/. Optionally add the address of your sitemap. The tool writes a correctly formatted robots.txt, with a User-agent line for each crawler followed by its rules, and a Sitemap line at the end. Copy it or download it.

Upload the file to the root of your domain so it is reachable at yourdomain.com/robots.txt. Robots.txt tells well-behaved crawlers which paths not to fetch, but it is not a security feature, since the file is public and bad bots ignore it, and a blocked page can still appear in search results if other sites link to it. To keep a page out of search results, use a noindex tag instead, and never block CSS or JavaScript files that search engines need to render your pages.

Practical Use Cases

  • Stop search engines from crawling admin, cart or internal search pages.
  • Point crawlers to your XML sitemap.
  • Create a starter robots.txt for a new website.
  • Block a specific crawler from one section while allowing others.

Tips for Best Results

  • Test your rules in Google Search Console's robots.txt report after you upload the file, since a typo can block a whole site.
  • Never block CSS, JavaScript or image folders that your pages need, because search engines use them to render and understand the page.
  • Add your sitemap line, and use noindex, not Disallow, for pages that you want kept out of search results.

Frequently Asked Questions

It is a plain text file in your site's root folder that tells search engine crawlers which paths they may or may not crawl. It is part of the Robots Exclusion Protocol.
No. Reputable crawlers such as Googlebot follow it, but malicious bots ignore it. Use authentication for pages that must stay private.
Not reliably. Disallow stops crawling, but a page can still be indexed from links. Use a noindex robots meta tag, and leave the page crawlable so Google can see the tag.
In the root of your domain, so it can be opened at yourdomain.com/robots.txt. Files in sub-folders are not used.
No. The file is generated in your browser and not sent anywhere.