SEO Tools

Free Robots.txt Generator Online

Create a valid robots.txt file with CMS templates, per-bot rules, AI crawler blocking and a live URL tester. Download or copy with one click.

⚡ CMS & Platform Presets

Crawler Rules

✓ Wildcard — applies to all crawlers · —

🧪 URL Tester

Test whether a URL path would be crawled or blocked by your current rules.

robots.txt

1 rules
# robots.txt — Generated by PursTech
# Created: 2026-05-05
# https://purstech.com/tools/robots-txt-generator

User-agent: *
Disallow: /admin/

📋 Deployment

Upload as robots.txt to your website root so it is accessible at yoursite.com/robots.txt

🤖 Bot Reference Guide

✓
Googlebot — Google's main search crawler
✓
Bingbot — Microsoft Bing search crawler
✓
Slurp — Yahoo search crawler
✓
DuckDuckBot — DuckDuckGo search crawler
⚠
GPTBot — OpenAI — trains ChatGPT on your content
⚠
ClaudeBot — Anthropic — trains Claude AI
⚠
CCBot — Common Crawl — used for AI training data
⚠
anthropic-ai — Anthropic AI data collection
⚠
Google-Extended — Google AI training (Gemini)
✓
AhrefsBot — SEO tool — backlink analysis crawler
✓
SemrushBot — SEMrush SEO tool crawler
✓
MJ12bot — Majestic SEO crawler
✓
DotBot — Moz SEO crawler

How to Use the Robots.txt Generator

1
Pick a preset
Select your CMS platform for instant pre-configured rules. Use 'Block AI Bots' to prevent AI training on your content.
2
Customise rules
Add, edit or remove crawl rules. Consult the Bot Reference Guide to understand what each user-agent does.
3
Test your URLs
Use the URL Tester to verify which paths would be crawled or blocked before you deploy your file.
4
Download & deploy
Click Download and upload robots.txt to your website root. Add your sitemap URL at the bottom for faster indexing.

❓ Frequently Asked Questions

What is a robots.txt file and why do I need one?+
A robots.txt file placed at your website root tells search engine crawlers which pages they can or cannot visit. Without one, crawlers index everything by default. A robots.txt lets you block admin pages, duplicate content and low-value URLs — improving crawl efficiency and preventing private pages from appearing in search results.
Does blocking a page in robots.txt prevent it from appearing in Google?+
Not always. robots.txt prevents crawlers from accessing page content, but Google can still index a URL if it appears in a link from another page — it just won't know the content. To fully prevent a URL from appearing in search results, use a 'noindex' meta tag or HTTP header instead. robots.txt and noindex serve different purposes and you can use both together.
Should I block AI bots like GPTBot and ClaudeBot?+
This is your choice as a content owner. AI bots like GPTBot (OpenAI) and ClaudeBot (Anthropic) crawl websites to train large language models. If you don't want your content used for AI training, disallow these bots in your robots.txt. Use our 'Block AI Bots' preset to block the most common AI training crawlers with one click.
What is Crawl-delay and should I use it?+
Crawl-delay tells a bot how many seconds to wait between page requests. This is useful for servers with limited resources that can't handle aggressive crawling. However, Google officially ignores Crawl-delay — it manages its own crawl rate based on your server's response times. Crawl-delay is respected by some other bots including Bingbot.
Where should I put my sitemap in robots.txt?+
Add a Sitemap directive at the bottom of your robots.txt file: Sitemap: https://yoursite.com/sitemap.xml. This helps all search engines discover your sitemap automatically. You can include multiple Sitemap lines for multiple sitemap files. This complements but does not replace submitting your sitemap directly in Google Search Console.