UFOZoo

Robots.txt Generator — Free Create & Test Rules Online

Generate robots.txt to control search crawlers. Set User-agent, Allow, Disallow, Crawl-delay, and Sitemap. All in your browser — no uploads.

Features

  • ✓Generate robots.txt files with multiple User-agent groups for different crawlers
  • ✓Add Allow and Disallow rules to control which paths crawlers can access
  • ✓Support for major search engine bots: Googlebot, Bingbot, Baiduspider, and all crawlers (*)
  • ✓AI crawler presets including GPTBot, ClaudeBot, Google-Extended, CCBot, and more
  • ✓Custom user-agent input for any crawler not covered by presets
  • ✓Pre-built templates for common scenarios: Standard, Block AI Training, WordPress, E-Commerce, Strict
  • ✓URL path tester — simulate access against your rules for any crawler
  • ✓Crawl-delay directive to control request rate and reduce server load
  • ✓Sitemap URL directives to help crawlers discover your content
  • ✓Real-time output preview as you build your rules — no Generate button needed
  • ✓One-click copy to clipboard with visual feedback
  • ✓Save robots.txt as a .txt file for direct deployment
  • ✓All processing runs entirely in your browser — zero server uploads, complete privacy
  • ✓Syntax-correct output formatted to official robots.txt standards (RFC 9309)
  • ✓Perfect for SEO audits, site launches, and staging environment configuration

How to Use

  1. 1Select a User-agent from the presets (Googlebot, Bingbot, Baiduspider, or any AI crawler) or enter a custom one.
  2. 2Enter the path you want to allow or block (e.g., /admin/, /private/, /public/).
  3. 3Choose Allow or Disallow — Disallow blocks the crawler from the path, Allow overrides a broader Disallow.
  4. 4Click 'Add Rule' to add the rule to your list. Repeat for as many rules as you need.
  5. 5Use the Templates tab to load preset configurations (Block AI Training, WordPress, E-Commerce, etc.).
  6. 6Use the Test URL tab to simulate whether a specific URL path is allowed or blocked for a given crawler.
  7. 7Optionally set a Crawl-delay value in seconds to throttle crawler request frequency.
  8. 8Add one or more Sitemap URLs pointing to your XML sitemaps.
  9. 9Copy the generated robots.txt and paste it into the root of your website, or save it as a .txt file.

Frequently Asked Questions

What is a robots.txt file?

∨
robots.txt is a plain text file placed in the root of a website that tells search engine crawlers which pages or sections they can or cannot crawl. It follows the Robots Exclusion Standard and uses directives like User-agent, Disallow, and Allow. When a crawler visits your site, it checks robots.txt first to determine which URLs to access.

Can I use wildcards and patterns in paths?

∨
Yes. The robots.txt standard supports wildcards: '*' matches any sequence of characters, and '$' matches the end of a URL. For example, 'Disallow: /*.pdf$' blocks all PDF files, and 'Disallow: /private/*' blocks everything under /private/. However, not all search engines support every pattern, so keep it simple when possible.

What is Crawl-delay and how does it work?

∨
Crawl-delay tells search engine crawlers to wait a specified number of seconds between requests. This is useful for reducing server load during crawls. Note that Googlebot ignores Crawl-delay — use Google Search Console's crawl rate settings instead. Other crawlers like Bingbot and Baiduspider may respect it.

Should I include Sitemap directives?

∨
Yes. Adding Sitemap directives in robots.txt helps crawlers discover your XML sitemaps faster. The sitemap URL should be the full, absolute URL (e.g., https://example.com/sitemap.xml). You can include multiple Sitemap directives for different sitemap files (pages, images, news, etc.).

How do I test if my robots.txt is working?

∨
Most search engines provide testing tools. Google Search Console has a robots.txt tester that shows how Googlebot interprets your file. You can also view your robots.txt directly by visiting https://yourdomain.com/robots.txt in a browser. The tool here includes a built-in URL tester — use the Test URL tab to simulate access for any crawler against your current rules.

Can robots.txt be used for security?

∨
No. robots.txt is a voluntary guideline for well-behaved crawlers — it does not enforce access restrictions. Malicious bots or scrapers will ignore it entirely. Never hide sensitive data behind a Disallow directive. Use proper authentication and server-side access controls for security.

Does robots.txt affect SEO rankings?

∨
Indirectly, yes. Properly configured robots.txt ensures search engines crawl and index your important pages while ignoring low-value sections (admin panels, search results, tag pages). Blocking important pages by mistake can prevent them from appearing in search results. Always double-check your rules before deploying.

What happens if I don't have a robots.txt file?

∨
Search engines will crawl your entire site without restriction. This is fine for most small sites, but larger sites benefit from using robots.txt to conserve crawl budget — directing crawlers away from duplicate, thin, or low-value pages toward your most important content.

What AI crawlers should I block?

∨
Popular AI training crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini/Bard), CCBot (Common Crawl), and Bytespider (ByteDance). If you do not want your content used for AI model training, add Disallow rules for these bots. AI search bots like ChatGPT-User and PerplexityBot may also be blocked if you prefer not to serve AI search tools. Use the Block AI Training template in this tool to quickly set up these rules.

How do I use the templates?

∨
Switch to the Templates tab to choose from pre-built configurations: Standard (allow all), Block AI Training (disallow AI training bots), WordPress (disable admin paths), E-Commerce (disable cart/checkout paths), and Strict (only allow homepage and sitemap). Each template populates the rules, crawl delay, and sitemaps automatically. You can then customize further in the Rules tab.

Related Tools

XML Sitemap Generator — Free Create Sitemaps for Google

Generate XML sitemaps for search engines. Add URLs with priority, change frequency, and last modified date. All processing runs in your browser.

Meta Tag Generator — Free Create SEO Title & Description

Create title tags, meta descriptions, Open Graph tags, Twitter Cards, and canonical URLs. Preview Google search results and social media appearance.

URL Slug Generator — Create SEO-Friendly URL Slugs Online

Convert any text into clean, SEO-friendly URL slugs. Lowercase, remove special characters, replace spaces with hyphens. All processing in your browser.