Skip to main content
UFOZoo

Robots.txt Generator: Create & Test Crawler Rules Online

Robots.txt generator online: create robots.txt rules with user agents, allow and disallow paths, crawl delay, and sitemap.

Updated 2026-08-26

Related Tools

XML Sitemap Generator: Create Sitemaps for Google Online

Meta Tag Generator: Create SEO Title & Description Online

URL Slug Generator: Create SEO-Friendly URL Slugs Online

Schema Markup Generator: Create JSON-LD for SEO Online

Adler-32 Checksum: zlib & PNG Fast Integrity Check

Alt Text Generator: Image Accessibility & WCAG Descriptions

Features

  • Generate robots.txt files with multiple User-agent groups for different crawlers
  • Add Allow and Disallow rules to control which paths crawlers can access
  • Support for major search engine bots: Googlebot, Bingbot, Baiduspider, and all crawlers (*)
  • AI crawler presets including GPTBot, ClaudeBot, Google-Extended, CCBot, and more
  • Custom user-agent input for any crawler not covered by presets
  • Pre-built templates for common scenarios: Standard, Block AI Training, WordPress, E-Commerce, Strict
  • URL path tester: simulate access against your rules for any crawler
  • Crawl-delay directive to control request rate and reduce server load
  • Sitemap URL directives to help crawlers discover your content
  • Real-time output preview as you build your rules: no Generate button needed
  • Copy or download the final file: one-click copy with visual feedback, or save as a .txt file for direct deployment
  • Syntax-correct output formatted to official robots.txt standards (RFC 9309)

How to Use

  1. 1Select a User-agent from the presets (Googlebot, Bingbot, Baiduspider, or any AI crawler) or enter a custom one.
  2. 2Enter the path you want to allow or block (e.g., /admin/, /private/, /public/).
  3. 3Choose Allow or Disallow: Disallow blocks the crawler from the path, Allow overrides a broader Disallow.
  4. 4Click 'Add Rule' to add the rule to your list. Repeat for as many rules as you need.
  5. 5Use the Templates tab to load preset configurations (Block AI Training, WordPress, E-Commerce, etc.).
  6. 6Use the Test URL tab to simulate whether a specific URL path is allowed or blocked for a given crawler.
  7. 7Optionally set a Crawl-delay value in seconds to throttle crawler request frequency.
  8. 8Add one or more Sitemap URLs pointing to your XML sitemaps.
  9. 9Copy the generated robots.txt and paste it into the root of your website, or save it as a .txt file.

Frequently Asked Questions

What is a robots.txt file?

robots.txt is a plain text file placed in the root of a website that tells search engine crawlers which pages or sections they can or cannot crawl. It follows the Robots Exclusion Standard and uses directives like User-agent, Disallow, and Allow. When a crawler visits your site, it checks robots.txt first to determine which URLs to access.

Can I use wildcards and patterns in paths?

Yes. The robots.txt standard supports wildcards: '*' matches any sequence of characters, and '$' matches the end of a URL. For example, 'Disallow: /*.pdf$' blocks all PDF files, and 'Disallow: /private/*' blocks everything under /private/. However, not all search engines support every pattern, so keep it simple when possible.

What is Crawl-delay and how does it work?

Crawl-delay tells search engine crawlers to wait a specified number of seconds between requests. This is useful for reducing server load during crawls. Note that Googlebot ignores Crawl-delay; use Google Search Console's crawl rate settings instead. Other crawlers like Bingbot and Baiduspider may respect it.

Should I include Sitemap directives?

Yes. Adding Sitemap directives in robots.txt helps crawlers discover your XML sitemaps faster. The sitemap URL should be the full, absolute URL (e.g., https://example.com/sitemap.xml). You can include multiple Sitemap directives for different sitemap files (pages, images, news, etc.).

How do I test if my robots.txt is working?

Most search engines provide testing tools. Google Search Console has a robots.txt tester that shows how Googlebot interprets your file. You can also view your robots.txt directly by visiting https://yourdomain.com/robots.txt in a browser. The tool here includes a built-in URL tester; use the Test URL tab to simulate access for any crawler against your current rules.

Can robots.txt be used for security?

No. robots.txt is a voluntary guideline for well-behaved crawlers; it does not enforce access restrictions. Malicious bots or scrapers will ignore it entirely. Never hide sensitive data behind a Disallow directive. Use proper authentication and server-side access controls for security.

Does robots.txt affect SEO rankings?

Indirectly, yes. Properly configured robots.txt ensures search engines crawl and index your important pages while ignoring low-value sections (admin panels, search results, tag pages). Blocking important pages by mistake can prevent them from appearing in search results. Always double-check your rules before deploying.

What happens if I don't have a robots.txt file?

Search engines will crawl your entire site without restriction. This is fine for most small sites, but larger sites benefit from using robots.txt to conserve crawl budget, directing crawlers away from duplicate, thin, or low-value pages toward your most important content.

What AI crawlers should I block?

Popular AI training crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini/Bard), CCBot (Common Crawl), and Bytespider (ByteDance). If you do not want your content used for AI model training, add Disallow rules for these bots. AI search bots like ChatGPT-User and PerplexityBot may also be blocked if you prefer not to serve AI search tools. Use the Block AI Training template in this tool to quickly set up these rules.

How do I use the templates?

Switch to the Templates tab to choose from pre-built configurations: Standard (allow all), Block AI Training (disallow AI training bots), WordPress (disable admin paths), E-Commerce (disable cart/checkout paths), and Strict (only allow homepage and sitemap). Each template populates the rules, crawl delay, and sitemaps automatically. You can then customize further in the Rules tab.

Why is my robots.txt not working?

Most often because the file is cached or the rule syntax is wrong. Check: the file must be at the domain root, each rule on its own line, and Disallow paths must match exactly. Google caches robots.txt for up to 24 hours, so changes take time. Test the result in Google Search Console under Robots.txt Tester.

What is the difference between robots.txt and meta robots noindex?

robots.txt blocks crawling (the bot never downloads the page), while meta robots noindex blocks indexing (the bot downloads but does not store it). If you use robots.txt to block a page, Google may still index the URL without content. For removing pages from search results, noindex is the correct tool.

How do I set up robots.txt on WordPress: do I create a physical file?

WordPress generates a virtual robots.txt on the fly, so you usually do not need a physical file at all. Without one, WP serves a default file that allows everything except /wp-admin/ and /wp-includes/. To customize, use an SEO plugin (Rank Math, Yoast, AIOSEO) or the robots_txt filter hook in your theme's functions.php. If you do upload a physical robots.txt, it overrides the virtual file entirely, which surprises many people when their plugin settings suddenly stop working.

I accidentally blocked my whole site with Disallow: /. How do I recover?

Fix the file first (remove the rule or narrow it to a real path). Search engines re-fetch robots.txt frequently; Google re-checks it at least every 24 hours, so recovery is usually a matter of hours, not days. Then use Search Console's URL Inspection tool to request indexing for your key pages. The real damage of a full block is not the paused crawling but deindexing: pages already in the index can drop out if Googlebot cannot fetch them for a long time, so fix it fast, and validate with this tool's URL tester before deploying next time.