Free online tool

Robots.txt Generator & Tester

Build a correct robots.txt in seconds. Choose which search engines, AI bots and SEO crawlers may visit, block private folders, add your sitemap — then test any URL.

Presets:

All robots (*)

Crawl-delay

Blocked folders / pages one per line

Allowed exceptions optional

Sitemap URLs one per line

robots.txt

Test a URL

Your rules are built in your browser. “Import” only reads the public robots.txt of the site you enter.

How it works

How to use the Robots.txt Generator

Pick your rules

Start from a preset, then allow or block each crawler and folder.

Test URLs

Check whether Googlebot or any other bot can crawl a specific page.

Download

Download robots.txt and upload it to the root of your website.

Features

Why use our Robots.txt Generator?

Smart presets

Allow all, WordPress, Laravel, or block AI training bots in one click.

25+ crawlers

Google, Bing, Yandex, GPTBot, ClaudeBot, CCBot, Ahrefs, Semrush and more.

Built-in tester

Test paths with Google’s matching rules, including * and $ wildcards.

Import live file

Load the current robots.txt from any website and edit it.

Sitemap lines

Add one or more sitemap URLs.

Safety warnings

Warns you before you accidentally block your whole site.

Works well with: schema markup generator, xml sitemap generator, keyword suggestion tool, serp preview — or browse all seo tools.

FAQ

Frequently asked questions

Everything you need to know about the Robots.txt Generator.

Back to the tool

What is robots.txt?

robots.txt is a text file at the root of your website (example.com/robots.txt) that tells crawlers which pages they may or may not visit.

Does robots.txt remove pages from Google?

No. Blocking a URL stops crawling, but the URL can still appear in results if other sites link to it. To remove a page, use a noindex meta tag and allow crawling.

How do I block AI bots like ChatGPT?

Use the “Block AI bots” preset. It disallows GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and other AI crawlers while keeping Google Search allowed. Well-behaved bots respect it.