Build a Valid Robots.txt Without Learning the Syntax

By ZeroFee Tools · October 4, 2026 · 7 min read

A robots.txt file is only a few lines of plain text, yet one wrong line can deindex an entire website. A stray Disallow: / tells every crawler to stay out, and the damage is silent until traffic collapses. The Robots.txt Generator builds the file correctly in seconds: create unlimited user-agent groups for Googlebot, Bingbot, AI crawlers, or all bots with the * wildcard, add allow and disallow path rules per group, and include the optional sitemap URL and crawl-delay fields. Everything is formatted to the robots.txt specification, ready to copy or download.

Site owners launching new projects use it to get the file right the first time, developers block staging and admin areas from crawlers, and SEOs audit client files for over-blocking mistakes. Upload the finished file to your domain root as robots.txt, then pair it with a proper sitemap built by the XML sitemap generator. If crawl errors reveal broken URL patterns, the double slash URL checker helps track them down.

What is a robots.txt file and where does it live?

A robots.txt file is a plain-text file at your domain root (example.com/robots.txt) that tells search engine crawlers which parts of your site they may or must not crawl. It uses User-agent lines to target bots and Allow/Disallow lines to set the rules for each group.

The file lives at a fixed address: your domain root, as in example.com/robots.txt. Crawlers check that location before they crawl anything else, which is why the file must sit at the root and not in a subdirectory. Its job is traffic control for bots: User-agent lines name which crawler a rule applies to, and Allow and Disallow lines say which paths that crawler may or must not visit.

Most sites need only a few lines. Allow everything by default, disallow admin panels, login pages, and staging subdomains, and point to the sitemap so crawlers find it immediately. Over-blocking is the classic mistake: site owners copy aggressive rules from forum posts and accidentally lock out the pages they want ranked. When in doubt, allow more and block less.

AI crawlers add a modern wrinkle. Training bots from AI companies increasingly respect robots.txt, so a disallow rule is currently the main lever site owners have for opting content out of training datasets. The generator's per-agent groups make it easy to address these crawlers separately from search engines.

Does blocking a page in robots.txt remove it from Google?

No. A Disallow rule only stops crawling; Google may still index the URL if other sites link to it. To truly remove a page from search results, use a noindex meta tag instead, and use robots.txt alongside it to save crawl budget.

This is the most misunderstood behavior in technical SEO. A Disallow rule is a crawling directive, not an indexing directive: it tells the crawler not to visit the URL, but Google can still index the URL itself if other sites link to it, showing it in results without a description. Blocking a page in robots.txt therefore never guarantees its removal from search results.

To truly remove a page, use a noindex meta tag, which tells Google to drop the URL from the index. The two mechanisms work best together: noindex removes the page from results while robots.txt keeps crawlers from wasting crawl budget on it. Just remember the ordering trap: if robots.txt blocks the crawler from visiting the page, it may never see the noindex tag, so apply noindex first and confirm removal before blocking the crawl.

Staging subdomains are the highest-risk area for robots.txt mistakes. A staging copy of the site should block all crawlers, while production should block almost nothing, and teams routinely deploy the wrong file to the wrong environment. Generate both versions with the tool, label them clearly, and verify the live file after every deploy.

What is crawl-delay and should you use it?

Crawl-delay asks polite crawlers to wait a set number of seconds between requests, which protects small servers from aggressive bots. Google ignores crawl-delay, so manage Googlebot's crawl rate in Google Search Console instead of relying on the directive.

Crawl-delay is a polite request, not an order. It asks well-behaved crawlers to wait a set number of seconds between requests, which protects small or shared servers from bots that hit dozens of pages per second. For a modest site on budget hosting, a small delay can be the difference between staying online and buckling under bot traffic.

The catch is that Google ignores crawl-delay entirely. If Googlebot is the crawler overwhelming your server, the directive does nothing, and you must manage its crawl rate in Google Search Console instead. Set crawl-delay for the smaller bots that respect it, handle Google separately, and treat the field as one layer of server protection rather than a complete solution.

How to use the Robots.txt Generator in 4 steps

1. Name the user-agent. Create a group for each crawler, such as Googlebot or Bingbot, or keep the * wildcard for all bots.
2. Add allow and disallow rules. Set path rules per group to control exactly which sections crawlers may visit.
3. Enter sitemap and crawl-delay. Add your sitemap URL and an optional crawl-delay for polite bots.
4. Copy, download, and upload. Copy or download the finished file and upload it to your domain root as robots.txt.

6 practical tips for safe crawl control

• Never put Disallow: / on a live site. One stray line blocks every crawler from everything; double-check the file before uploading.
• Keep the file minimal. Allow everything, block only admin, login, and staging areas, and point to the sitemap.
• Test after uploading. Use the robots.txt tester in Google Search Console to confirm the file parses as intended.
• Use noindex for real removals. Robots.txt saves crawl budget; only a noindex tag actually drops a URL from results.
• Reference your sitemap. A sitemap line in robots.txt helps crawlers discover it even before you submit it in Search Console.
• Audit for over-blocking yearly. Rules copied from old tutorials often block CSS, JS, or API paths that modern crawlers need to render pages.

Frequently asked questions