A common robots.txt mistake is thinking it hides pages from Google. It doesn't. A robots.txt file only asks bots not to visit a page. If other sites link to that page, Google can still list it in search results. Use robots.txt to manage crawling. When you want a page out of search, use a noindex tag or a password.
So how do you create a robots.txt file? Write a few short rules in a plain text editor, save the file as robots.txt, upload it to the top folder of your website and test it. That's the whole job. Want to skip the typing? Our free Robots.txt Generator builds the file in your browser. There's no sign-up, and it warns you before you block your whole site by accident.
Quick verdict: For most small business sites, a generator is the safest choice. It writes correct rules and lets you test them before you upload anything. WordPress users can paste the result into their SEO plugin's file editor. Writing the file by hand works too, but only if you know the syntax well.
What is a robots.txt file and what does it do?
A robots.txt file is a small text file at yourdomain.com/robots.txt. It tells crawlers which parts of your site they may visit. A crawler, also called a bot or spider, is a program that visits web pages on its own. Googlebot and Bingbot are the best known.
The rules follow the Robots Exclusion Protocol. It started in 1994 and became an official internet standard in September 2022 as RFC 9309, the Robots Exclusion Protocol standard.
Crawling vs indexing in plain words
Crawling means a bot visits a page and reads it. Indexing means the search engine saves the page so it can appear in results. Robots.txt only controls crawling.
Google and Bing describe the result differently. Google says a blocked URL can still appear in results if other pages link to it. Bing's help pages say a page Bing doesn't crawl won't be indexed. Either way, robots.txt is the wrong tool for removing a page from search.
Do you need one?
No, it's optional. Without a robots.txt file, bots assume they can crawl everything, and your site works normally. Still, a short robots.txt for a small business website is worth the five minutes. It keeps bots away from pages like your cart or login screen and points them to your sitemap.
A sitemap is a file, often called sitemap.xml, that lists the pages you want search engines to find.
Three ways to make a robots.txt file, compared
You can write the file by hand, edit it inside your website platform, or build it with a free generator. All three cost nothing. The real difference is how easy each one makes it to slip up.
| Method | Cost level | Skill needed | Risk of errors | Best for |
|---|---|---|---|---|
| Plain text editor (Notepad, TextEdit) | Free | Medium. You must know the syntax. | Higher. Typos are easy to miss. | Developers and custom sites |
| CMS plugin or platform settings (WordPress SEO plugins, store builders) | Free or included | Low | Medium. Plugins may add their own rules. | WordPress and hosted store owners |
| Free online generator (our Robots.txt Generator) | Free, no sign-up | Low | Lower. It includes presets, a tester and safety warnings. | Small businesses, bloggers, freelancers |
Plain text editor
Pros: full control, no extra software, works on any server. Cons: one wrong character can break a rule, and nothing warns you.
Google's guide to creating a robots.txt file says almost any text editor works. It warns against word processors like Word, because they can add curly quotes. Save the file as UTF-8, a standard plain text format with no styling.
CMS plugin or platform settings
Pros: you never touch a server folder. Many WordPress SEO plugins include a robots.txt editor. Cons: you're editing inside someone else's system, so plugins can add or change rules.
For example, you might save three clean lines, and the live file shows five. Always open /robots.txt in your browser after you save. On hosted platforms such as Shopify, follow the platform's own help pages before you change anything.
Free online generator
Pros: ready presets, correct syntax and a built-in URL tester. Cons: you still upload the file yourself, unless your platform has an editor where you can paste it. For a first robots.txt, this is the method we'd pick. It's the easiest way to avoid the classic "I blocked my whole site" accident.
Which should you choose?
- If you run WordPress with an SEO plugin: paste the generator output into the plugin's editor.
- If you have FTP or file manager access (Laravel, PHP or custom sites): generate the file, download it and upload it.
- If you use a hosted store or site builder: use the platform's own settings, and only change rules for a clear reason.
- If you know the syntax well: write it by hand, then check it with a tester.
How to create a robots.txt file in 4 steps
Google's official process has four steps: create the file, add rules, upload it to the root and test it. Here's what each one looks like in practice.
Step 1: Check if you already have one
Type yourdomain.com/robots.txt into your browser. If you see text, a file already exists. WordPress shows a virtual file by default, so this is common. Edit what's there instead of starting from zero. If you upload your own file to WordPress, it replaces the virtual one.
Step 2: Write the rules in a plain text editor
Open Notepad on Windows. On a Mac, open TextEdit and switch it to plain text mode. Type your rules and save the file as robots.txt in UTF-8, all lowercase. A site can have only one robots.txt file.
Step 3: Upload it to the root of your site
The root is the top folder of your domain. On most hosting, that's public_html. On Laravel sites, it's usually the public folder. In the right place, the file opens at https://yourdomain.com/robots.txt.
Each host needs its own file. blog.example.com is a different host from example.com. The http and https versions count as separate, and so do www and non-www.
Step 4: Test it
Open the live URL to confirm it loads. Then check that Googlebot can crawl your most important pages. Use the tester inside our generator or the separate Robots.txt Tester. Google Search Console, Google's free tool that shows how Google sees your site, is another place to check.
Robots.txt syntax, explained simply
Most files use four commands: User-agent, Disallow, Allow and Sitemap. Some add a fifth, Crawl-delay.
User-agent
This line names the bot that the rules below it apply to. User-agent: * means all bots. Every group of rules starts with a User-agent line.
Bing has one catch. According to the Bing Webmaster Tools robots.txt help, Bingbot ignores the general * group if you give it a group of its own. So copy your general rules into the Bingbot group too.
Disallow and Allow
Disallow means "don't visit this path." Allow makes an exception, such as one page inside a blocked folder. Paths are case-sensitive, so /Admin/ and /admin/ are different paths.
Sitemap
Add the full URL of your sitemap, like Sitemap: https://www.example.com/sitemap.xml. You can list more than one. No sitemap yet? Our free XML Sitemap Generator can make one.
Wildcards * and $
The * sign matches any characters. The $ sign marks the end of a URL. So Disallow: /*.pdf$ blocks every URL that ends in .pdf.
Crawl-delay
This line asks bots to wait a set number of seconds between visits. Bing, Yahoo and Yandex follow it. Google ignores it.
Robots.txt example files you can copy
Each robots.txt example below is complete. Swap example.com for your own domain and keep only the rules you need.
Allow everything
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
An empty Disallow line blocks nothing. Many small sites need nothing more.
Block one folder
User-agent: *
Disallow: /login/
The trailing slash matters. Disallow: /login would also block /login-help, because rules match the start of a path.
Block one page or file type
User-agent: *
Disallow: /thank-you/
Disallow: /*.pdf$
This keeps bots off your thank-you page and every PDF file on the site.
Block one bot
User-agent: GPTBot
Disallow: /
This asks OpenAI's GPTBot to stay off the whole site. Every other bot is unaffected.
Allow one page inside a blocked folder (WordPress)
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap.xml
Many WordPress themes need admin-ajax.php to work. That's why the Allow line stays.
What should you block in robots.txt?
Block pages with no search value: admin and login areas, cart and checkout, thank-you pages, internal search results and endless filter URLs. Leave open everything a customer might search for.
Decision table by site type
| Area | Example path | Common on | What to do |
|---|---|---|---|
| Admin or login | /wp-admin/, /admin/, /login/ | WordPress, Laravel, all sites | Block, and keep it behind a password |
| Cart and checkout | /cart/, /checkout/ | Online stores | Block |
| Thank-you pages | /thank-you/ | Service businesses (plumbers, dentists) | Block |
| Internal search results | /?s=, /search/ | Blogs, stores | Block |
| Filter and sort URLs | /*?sort=, /*?color= | Online stores | Block if they create many near-copy pages |
| Staging site | staging.example.com | Any site in development | Block on staging only, plus a password |
| CSS, JavaScript, images | /assets/, /css/ | All sites | Leave open |
| Old page you want gone | /old-offer/ | All sites | Use noindex, don't block |
Picture a local dentist's website. A sensible file blocks /thank-you/, /login/ and internal search, then stops. Service pages, location pages and the blog all stay open.
What not to block
Don't block CSS or JavaScript files. Google needs them to see your page the way visitors do. Don't block important pages "just in case." And don't block a page you want removed from Google, because Google then can't read the noindex tag on it.
Why robots.txt can't hide private files
Anyone can read your robots.txt file. RFC 9309 says its rules are not a form of access control, and listing a path makes it visible to anyone who opens the file. Writing Disallow: /client-invoices/ is like putting up a sign that points to the folder. Private content needs a login or password.
How to use our free robots.txt generator
Our robots.txt generator is free and builds a correct file in a few clicks. It works in your browser, so you can test your rules before anything goes live.
Pick a preset
Open the Robots.txt Generator and choose a starting point. The presets are Allow all, Block AI bots, WordPress, Laravel and Block everything. Use Block everything on a staging site only.
Choose crawlers, folders and sitemap
- Set All robots (*) to Allowed or Blocked.
- Adjust single crawlers from the bot list, which includes Google, Bing, GPTBot and ClaudeBot.
- Pick a Crawl-delay, or leave it at None.
- Type paths into Blocked folders / pages, one per line.
- Add any Allowed exceptions.
- Enter your Sitemap URLs.
Test a URL
Use Test a URL to check whether Googlebot or another bot can crawl a specific page. The tester follows Google's matching rules, including the * and $ wildcards. Test your homepage and your main service page first. If either shows as blocked, fix the rules before you go further.
Download and upload
Click Copy or Download. For a small tweak, choose Edit manually. Upload the file to your root folder or paste the text into your plugin's editor. Then open yourdomain.com/robots.txt to confirm it's live.
Import your live file to edit it
Already have a file? Click Import live robots.txt and enter your domain. The tool reads only the public file, the same one anyone can open in a browser. Your rules are built in your browser.
Should you block AI bots?
It depends on what you want. Blocking AI bots can keep your content out of some AI training. It may also make your site less likely to appear in AI tools that rely on those bots.
What the Block AI bots preset does
The preset adds Disallow rules for GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and other AI crawlers. Google Search stays allowed. Google-Extended only controls whether Google may use your content to train its Gemini AI models. It has no effect on regular Google Search.
Why it's a request, not a lock
Robots.txt can only ask. Well-behaved bots follow it, but nothing forces a bot to obey. Our view: block AI bots if you sell original content and don't want it used for training. If you'd like to be quoted in AI answers, think twice before blocking them all.
Robots.txt vs noindex vs password: which one to use
Start with your goal. Robots.txt controls crawling, noindex controls search listings, and a password controls who can see the page at all.
| Your goal | Right tool | Note |
|---|---|---|
| Stop bots from crawling low-value pages | robots.txt | The page can still be listed if others link to it |
| Remove a page from Google | noindex meta tag, with crawling allowed | Google must crawl the page to see the tag |
| Keep a page private | Password or login | The only option that actually stops people |
| Ask AI bots not to train on your content | robots.txt | Bots follow it only if they choose to |
A noindex tag is a short line in a page's code that tells search engines not to list it. You can create one with our Meta Tag Generator. Since September 2019, Google no longer supports noindex lines inside robots.txt, so the tag must go on the page itself.
What does "URL blocked by robots.txt" mean?
It means Google found a URL, but your robots.txt file stops Googlebot from crawling it. You'll usually see this status in Google Search Console.
The fix depends on what you want:
- The page should rank: find the Disallow rule that matches it, remove or narrow it, and test the URL again.
- The page should leave Google: remove the block and add a noindex tag instead.
- The page doesn't matter (a cart or login page, for example): ignore the warning.
Common robots.txt mistakes and how to fix them
Most problems come from a few repeat errors. Each takes minutes to fix once you spot it.
Disallow: / after launch
Developers often block a staging site with Disallow: /. If that file gets copied to the live site, the site can start dropping out of Google. Check /robots.txt after every launch, redesign or host move.
Blocking CSS and JS
Old advice told people to block theme and script folders. Remove those rules so Google can render your pages the way visitors see them.
Noindex plus Disallow
If a page is blocked, Google can't read its noindex tag. Pick one approach. To remove a page, use noindex and allow crawling.
Crawl-delay and Google
A crawl-delay line for Googlebot does nothing, because Google ignores it. It only affects bots such as Bingbot and Yandex.
Wrong location, name or format
The file must sit at the root, be named robots.txt in lowercase and be saved as plain UTF-8 text. A file at /files/robots.txt is ignored. So is one typed in Google Docs and saved with formatting.
Size matters too. Per how Google reads robots.txt, Google ignores anything past 500 KiB, and RFC 9309 says crawlers must read at least that much. A small business file should be far shorter.
Old sitemap line
Switching SEO plugins or moving to a new site can change your sitemap URL. Update the Sitemap line so it doesn't point to a missing file.
Get your robots.txt and SEO checked
Once your file is live, run a full check. Our free Website SEO Checker scores any page and lists clear fixes. You can rebuild or test your rules anytime with the Robots.txt Generator.
If robots.txt, sitemaps and crawl errors feel risky, our SEO services team can handle them. We build SEO and performance into every site from day one, including technical SEO setup. Every project gets a dedicated team, a project manager and weekly demos. After launch, our maintenance plans cover updates, security patches, backups and performance monitoring.
We serve clients worldwide and reply within one business day. You get a fixed quote after a free discovery call, with no hidden costs. Request a Free Quote to get started.
Frequently Asked Questions
What is a robots.txt file?
It's a plain text file at the root of your website that tells bots which pages they may visit. It controls crawling, not whether a page appears in search.
How do I create a robots.txt file for a WordPress website?
Use your SEO plugin's robots.txt editor, or upload your own file to the root folder. Keep the Allow: /wp-admin/admin-ajax.php line. Our generator's WordPress preset adds it for you.
How do I create a robots.txt file in Shopify?
First, visit yourstore.com/robots.txt to see what's already there. Hosted platforms like Shopify manage files inside their own system, so you can't always upload one the usual way. Follow Shopify's help pages before you change any rules.
Does robots.txt remove a page from Google?
No. A blocked page can still appear if other sites link to it. To remove a page, add a noindex tag and let Google crawl it.
How do I block ChatGPT and other AI bots?
Add a Disallow rule for GPTBot and other AI crawlers, or use the Block AI bots preset in our generator. Only bots that choose to follow robots.txt will listen.
Does Google support crawl-delay?
No, Google ignores crawl-delay. Bing, Yahoo and Yandex follow it.
Are my rules safe in your tool?
Yes. Your rules are built in your browser. The Import feature only reads the public robots.txt file of the site you enter.
