Robots.txt Generator & Validator

Generate, customize, and validate RFC 9309-compliant robots.txt files. Manage search bots, block AI scrapers, configure sitemaps, and test URL access in real-time.

Robots.txt Generator & Validator

Create, customize, and validate RFC 9309-compliant robots.txt files for your website. Control search engine crawlers, block AI web scrapers, configure XML sitemaps, and test URL access in real-time.

Quick Presets:

Crawler Directives

Sitemaps & Host

AI & LLM Training Crawler Shield

Prevent your content from being scraped by AI training models and commercial LLM bots.

GPTBotChatGPT-UserClaudeBotCCBotPerplexityBotBytespiderGoogle-ExtendedAmazonbot

robots.txt Preview

Syntax DiagnosticsValid RFC 9309
No syntax errors found. All paths conform to Robots Exclusion Protocol standards.

Interactive URL Tester & Access Sandbox

Test any URL path against your generated rules to verify whether a crawler is allowed or blocked.

https://example.com
BLOCKED
Evaluated locally
Matched Directive: Disallow: /admin/

The crawler Googlebot will be barred from crawling this path because the rule Disallow: /admin/ matched with highest precedence.

Understanding Robots.txt and the Robots Exclusion Protocol

The robots.txt file is a fundamental standard of the modern World Wide Web. Placed at the root directory of a domain (for example, https://example.com/robots.txt), it instructs automated web robots, search engine spiders (such as Googlebot, Bingbot, and Baiduspider), and artificial intelligence training crawlers (such as GPTBot, ClaudeBot, and CCBot) which pages, paths, or directories they are permitted or barred from requesting.

In September 2022, the Internet Engineering Task Force (IETF) formalized the standard under RFC 9309 (Robots Exclusion Protocol). This specification unified syntax rules, longest-match precedence mechanics, and pattern handling across all compliant search engines and scraping bots.

The Robots.txt Generator & Validator provides a visual interface to build, customize, lint, and test your crawler directives directly in your browser without requiring manual syntax memorization.

Core Robots.txt Directives Explained

A valid robots.txt file is structured into records. Each record begins with one or more User-agent directives, followed by path directives (Allow and Disallow), optional crawling rate controls (Crawl-delay), and global XML sitemap references.

1. User-agent

Specifies the specific crawler or robot the following directives apply to.

  • User-agent: * targets all crawlers globally unless a more specific record exists for a particular bot.
  • User-agent: Googlebot applies exclusively to Google’s primary search spider.
  • User-agent: GPTBot applies exclusively to OpenAI’s web crawler.

2. Disallow

Tells the crawler which URL paths must not be accessed.

  • Disallow: /admin/ prevents bots from crawling any URL that begins with /admin/.
  • Disallow: / disallows the entire website (commonly used on staging and private development environments).
  • Disallow: (with an empty path) explicitly allows crawling across all pages.

3. Allow

Explicitly grants crawler access to a specific sub-path or file within an otherwise disallowed directory.

  • For example, WordPress sites frequently disallow /wp-admin/ while explicitly allowing /wp-admin/admin-ajax.php so search engines can execute dynamic site elements.

4. Sitemap

Provides search engines with the absolute URL to your XML sitemaps to accelerate page discovery and indexation.

  • Sitemap: https://example.com/sitemap.xml
  • You can declare multiple Sitemap: directives if your platform uses sitemap indexes or split feeds.

5. Crawl-delay

Requests that crawlers wait a specified number of seconds between consecutive page requests to reduce server load.

  • Supported by Bingbot, Yahoo! Slurp, and Yandex. Googlebot ignores this directive and instead provides crawling speed controls in Google Search Console.

Pattern Matching with Wildcards and End Anchors

RFC 9309 supports two special characters for fine-grained URL path matching:

  • The Wildcard (*): Matches zero or more arbitrary characters.
    • Disallow: /*?*sort= blocks any URL query string containing the sorting parameter sort=.
    • Disallow: /*.pdf$ blocks all PDF files regardless of which subdirectory they reside in.
  • The End-of-String Anchor ($): Designates that the pattern must match up to the very end of the URL.
    • Disallow: /private$ matches exactly /private, but permits access to /private/team or /private-notes.

RFC 9309 Rule Precedence & Longest-Match Evaluation

When both Allow and Disallow directives match a particular URL, compliant crawlers resolve conflicts using longest-match specificity:

  1. The crawler compares the character length of all matching path patterns.
  2. The directive with the greatest number of matching characters wins.
  3. If both an Allow and a Disallow rule match with the exact same length, the Allow directive takes precedence.

Example Precedence Scenario:

User-agent: *
Disallow: /store/checkout/
Allow: /store/checkout/confirmation
  • Crawling /store/checkout/cart matches Disallow: /store/checkout/ (16 characters) and is BLOCKED.
  • Crawling /store/checkout/confirmation matches both rules. However, Allow: /store/checkout/confirmation is 28 characters long, whereas the Disallow rule is only 16 characters. Because the Allow directive is longer and more specific, the confirmation page is ALLOWED.

Best Practices for Modern Robots.txt Configuration

  1. Never use robots.txt to conceal private or sensitive data: A robots.txt file is completely public and viewable by any user or attacker. Do not list private administration paths containing secret tokens. Use HTTP authentication, IP allowlists, or session logins for security.
  2. Do not block CSS, JavaScript, or theme assets: Search engines render web pages like modern browsers to evaluate layout, mobile-friendliness, and Core Web Vitals. Blocking your styles or script folders can cause pages to be misclassified or lose rankings.
  3. Always use leading forward slashes (/): Path patterns must begin with / or * to conform to RFC 9309.
  4. Guard against unwanted AI scraping: If you do not wish your site’s content to be ingested for AI training, declare explicit Disallow rules for dedicated AI bots such as GPTBot, ClaudeBot, CCBot, and Bytespider.

Frequently Asked Questions (FAQ)

Where should the robots.txt file be uploaded?

Your robots.txt file must always be placed at the root level of your website domain, accessible at https://yourdomain.com/robots.txt. Placing it in a subfolder (such as https://yourdomain.com/assets/robots.txt) will cause crawlers to ignore it completely.

Does robots.txt prevent a page from appearing in Google Search results?

Not necessarily. While robots.txt instructs crawlers not to download or visit a page, Google may still index the URL if other public websites link to it. If you want to ensure a page never appears in search engine results, allow the bot to crawl it and add a <meta name="robots" content="noindex" /> tag to the page HTML or send an X-Robots-Tag: noindex HTTP response header.

How does this tool validate robots.txt rules?

The built-in Interactive URL Tester runs an RFC 9309-compliant pattern matching algorithm directly inside your browser. It evaluates wildcard symbols (*), end-anchors ($), and longest-match specificity to provide an instant verdict on whether a given path is allowed or blocked for any selected bot.

Are my custom paths or domain names stored on any server?

No. Processes locally in your browser — no files, code, paths, or URLs are uploaded to or stored on DwellixTools servers.