Robots.txt Generator — Create a Valid robots.txt Online Free

Understanding robots.txt

What robots.txt does

A robots.txt file sits at the root of your domain (https://example.com/robots.txt) and tells well-behaved crawlers — Googlebot, Bingbot, and the rest — which URL paths they may or may not fetch. It is the first thing most crawlers read when they visit a site.

The file is organized into groups: each group starts with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow rules for those crawlers. A Sitemap line at the end points crawlers at your XML sitemap.

Allow vs Disallow

• Disallow: /path — crawlers must not fetch URLs under that path.
• Allow: /path — explicitly permits a path (useful to re-allow a subfolder inside a disallowed directory).
• Disallow: (empty value) — allows everything for that user-agent.
• When two rules match the same URL, Google applies the most specific (longest) match.

Common patterns

Sitemap and crawl-delay

The Sitemap directive is optional but recommended: it helps crawlers discover your sitemap even before they find links to it. List it once, at the end of the file, with the full https URL.

Crawl-delay asks crawlers to wait N seconds between requests. It is supported by Bing and Yandex but ignored by Google — if Google crawls too aggressively, use the rate-limit settings in Google Search Console instead.

What robots.txt cannot do

robots.txt controls crawling, not indexing. A disallowed page can still appear in search results if other sites link to it, and Google may index the URL without its content. To keep a page out of search results entirely, use a noindex meta tag or require authentication — and remember crawlers must be allowed to fetch a page in order to see its noindex tag.