Want a Custom tool for Yourself?

Need a Custom Tool? We build custom tools that can save hours per employee per day.

Robots.txt Generator

Build a robots.txt file quickly: allow/disallow rules, crawl-delay, and sitemap links. Copy the result and publish it at /robots.txt.

Enter the Details




Search Engine Robots
















Restricted Directories

The root-relative path should end with a forward slash "/" to be valid.



GENERATED robots.txt




Last updated: February 18, 2026

Created by: Eon Tools Dev Team

Reviewed by: Prabindra Tamang


What this tool does

A robots.txt file tells search engine crawlers which parts of your site they may visit. This tool builds that file from a few choices, so you can set the default rule, point crawlers to your sitemap, and adjust how they treat your site, then copy the result and upload it.

How to use it

  1. Set the default for all robots: allow them everywhere, or disallow everything.
  2. Optionally choose a Crawl-Delay and add your Sitemap URL, then set any rules for specific crawlers.
  3. Generate, copy the file, and upload it to the root of your domain so it sits at yourdomain.com/robots.txt.

How it works

The tool writes the file in the standard format crawlers expect. Each block starts with a User-agent line naming the crawler, followed by Disallow (or allow) rules for the paths it covers, with your Crawl-delay and a Sitemap line added where you set them. Crawlers read this file on nearly every first visit to decide where they are allowed to go.

What robots.txt does, and does not, do

This is the part that catches people out, so it is worth stating plainly: robots.txt controls crawling, not indexing. Blocking a page stops crawlers from fetching it, but the page can still appear in Google if other sites link to it, shown as a bare URL with no description. If your goal is to keep a page out of search results, do the opposite of blocking it: allow it to be crawled and add a noindex tag, which the meta tag generator can produce. Block it in robots.txt and Google can never read that noindex, so it stays indexable.

Two more things worth knowing. First, robots.txt is not a security tool. The file is public, anyone can read it at your domain, so listing a private folder there actually advertises its location. Protect sensitive pages with a password or login, not a Disallow line. Second, Google ignores the Crawl-delay directive entirely; if you need to slow Google down, do it in Search Console, though some other crawlers do respect the setting. Used well, robots.txt is mainly about crawl budget on large sites, steering crawlers away from low-value URLs so your important pages get the attention. The SEO audit checklist covers where this fits in a full review.

Questions people ask

Does robots.txt stop a page from being indexed?

No. It stops crawling, not indexing. A blocked page can still show up in search if other pages link to it. To keep a page out of results, allow crawling and use a noindex tag instead.

Where does the robots.txt file go?

At the root of your domain, so it is reachable at yourdomain.com/robots.txt. Crawlers look for it there, and nowhere else.

Can I use robots.txt to hide private pages?

No, and you should not try. The file is public, so listing sensitive paths reveals them. Use proper authentication, such as a password or login, to protect private content.

Does Google obey the crawl-delay setting?

No. Google ignores crawl-delay and is controlled through Search Console instead. Some other crawlers, such as Bing, do honour it, which is why the option is there.

References

  1. Google Search Central. Introduction to robots.txt. https://developers.google.com/search/docs/crawling-indexing/robots/intro

Prabindra Tamang

Prabindra Tamang works in digital marketing and business development, with experience across campaigns, communications, and event execution. His strengths are in how digital content is presented, discovered, and made useful in practice. At Eon Tools, he reviews SEO tools.

Other Tools