Robots.txt Generator — Free Online Tool

How to Create a Robots.txt File

Select a preset or build custom rules by adding user-agent directives. For each bot (user-agent), specify which paths to allow or disallow. Add your sitemap URL so search engines can find your XML sitemap. Click "Generate" to get the complete robots.txt file — copy it and upload to your website's root directory (e.g., example.com/robots.txt). The file must be at the root level to work.

Why Robots.txt Matters for SEO

The robots.txt file tells search engine crawlers which pages they can and cannot access on your site. A properly configured robots.txt prevents indexing of duplicate content, admin pages, staging environments, and private areas. It also directs crawlers to your sitemap for efficient discovery of all your pages. Without it, bots may crawl and index pages you do not want in search results.

How to Block AI Crawlers in 2026

AI companies (OpenAI, Google, Anthropic, Meta, Apple) use web crawlers to scrape content for training data. This generator includes a "Block AI Crawlers" preset that blocks GPTBot, Google-Extended, ClaudeBot, FacebookBot, Applebot-Extended, CCBot, and other AI-specific user agents. This does not affect regular search indexing — Google's main crawler (Googlebot) remains allowed. Note that respecting robots.txt is voluntary — not all AI crawlers honor it.

Common Robots.txt Mistakes to Avoid

Do not block your CSS and JavaScript files — Google needs them to render your pages properly. Do not block your images folder if you want image search traffic. Always include a trailing slash for directories (e.g., /admin/ not /admin). Use Disallow: / carefully — it blocks the entire site for that user-agent. Test your robots.txt using Google Search Console's robots.txt tester before deploying.