robots.txt Generator - Build the Rules and Test a Path (Free)

Build the rules

User-agent (one crawler per line)

Disallow (paths to keep crawlers out of, one per line)

Allow (exceptions inside a disallowed path)

Sitemap (absolute URL, one per line)

Crawl-delay (seconds, leave blank to omit)

Test this path

Block every crawler from the whole site (Disallow: /)

robots.txt controls crawling, not indexing. A disallowed URL can still appear in search results as a bare URL if other sites link to it. To keep a page out of results, use a noindex meta tag and do not disallow that page, because a crawler that is blocked can never read the noindex.

Your robots.txt

Output

Lines: 0 Bytes: 0

Result for the tested path

—

Things worth checking

Frequently asked questions

If both a Disallow and an Allow rule match, which one wins?

The rule with the longer matching path wins. When both are the same length, Allow wins. This tool follows the same rule, so with Disallow: /admin/ and Allow: /admin/help/ the path /admin/help/index.html is allowed.

Can robots.txt remove a page from search results?

No. robots.txt only stops crawling. If other sites link to the URL it can still appear in results as a bare URL with no description. To keep a page out of results use a noindex meta tag and do not disallow that page, because a blocked crawler can never read the noindex.

Can I use * and $ in a path?

Yes. * matches any sequence and $ anchors the match to the end of the URL. For example Disallow: /*?sessionid= matches URLs with that query, and Disallow: /*.pdf$ matches URLs ending in .pdf. This tool supports both when it tests a path.

Should I set Crawl-delay?

Google ignores Crawl-delay. To reduce Google crawl rate, use Search Console instead. Some other crawlers such as Bing do read it, so add it only when you are targeting those.

Where does robots.txt go?

At the root of the site, for example https://example.com/robots.txt. It is only read at the domain root, so a copy in a subdirectory has no effect, and each subdomain needs its own file.

Is anything I type sent to a server?

No. Both the generation and the path test happen entirely in your browser. The paths and URLs you type are never sent to or stored on a server.

How to use

Choose the crawlers — Put the target in the User-agent box. Leave it as * to cover every crawler.
List paths to block — Write one path per line in the Disallow box. Each must start with /, and * and $ are supported.
Add exceptions — If something inside a disallowed directory should stay visible, add it to the Allow box. The longer match wins.
Add your sitemap — Write the absolute URL of sitemap.xml. Relative paths are not read, so it must start with https://.
Test a path — Type a real path into the test box to see whether the rules you just wrote would let a crawler fetch it.
Save and deploy — Download the robots.txt and place it at the root of your site.