Quick answer: A robots.txt file sits at the top level of your site and asks crawlers which paths they may visit, using User-agent, Allow and Disallow lines and a Sitemap line. It is a request that well-behaved crawlers follow, not a lock. It does not hide a page, and a blocked page can still appear in search results if other sites link to it.
A robots.txt file sits at the root of your website and tells crawlers, such as Googlebot and Bingbot, which parts of the site they may visit. It is often used to keep them out of admin areas, shopping carts and search results, and to point them to your sitemap.
Choose your rules and the file is written for you as you go. You can also block the crawlers that AI companies use to collect web content, and then copy the text or download it as robots.txt.
How to use the file
- Set your rules, then download the file or copy the text.
- Save it as
robots.txt(lowercase) and upload it to the top level of your site, so it opens athttps://yourdomain.com/robots.txt. - Open that address in a browser to check it. In Google Search Console you can also test it.
Things to know
- It is a request, not a lock. Well-behaved crawlers follow it, but it does not hide anything. Never use it to protect private pages. Use a password or login for that.
- Blocking is not the same as removing. A page blocked in robots.txt can still appear in search results if other sites link to it. To keep a page out of Google, use a
noindexmeta tag and let the page be crawled. - Do not block your CSS and JavaScript, as Google needs them to display and understand your pages.
- AI crawlers. Blocking a crawler such as GPTBot or Google-Extended asks that company not to use your content for training. Google-Extended does not affect normal Google Search results.
- “Block the whole site” is only for sites you do not want found, such as a private test version.
A worked example
Example. To let search engines crawl everything except your admin and cart areas, and to point them at your sitemap, the file is:
User-agent: * Disallow: /admin/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml
The main directives
| Line | Meaning | Example |
|---|---|---|
| User-agent | Which crawler the rules apply to | User-agent: * (all crawlers) |
| Disallow | A path not to crawl | Disallow: /admin/ |
| Allow | An exception inside a blocked path | Allow: /admin/public/ |
| Sitemap | The full address of your sitemap | Sitemap: https://yoursite.com/sitemap.xml |
| Crawl-delay | A pause between requests (Google ignores it) | Crawl-delay: 10 |
| * and $ | Wildcard and end of address | Disallow: /*.pdf$ |
Each group starts with a User-agent line. The file must be called robots.txt and open at the root of your domain.
Common mistakes to avoid
- Blocking pages you want to rank. A Disallow rule stops crawling, so a page you block cannot be read at all.
- Using robots.txt to hide private pages. Anyone can read the file, so protect private pages with a login instead.
- Blocking CSS and JavaScript files. Google needs them to display and understand your pages.
Go deeper: read our guide robots.txt and Meta Tags: A Practical SEO Guide.
Frequently asked questions
Where do I put the robots.txt file?
In the root folder of your website, so it opens at yourdomain.com/robots.txt. It will not work in a subfolder.
Does robots.txt stop a page appearing in Google?
Not reliably. It stops crawlers visiting the page, but the address can still be listed if other sites link to it. Use a noindex meta tag to keep a page out of search results.
Should I block AI bots?
It is your choice. Blocking asks AI companies not to collect your content for training. It will not remove content already collected, and only crawlers that respect robots.txt will obey. Blocking Google-Extended does not change your normal Google Search ranking.
What does Disallow: with nothing after it mean?
It means nothing is blocked, so crawlers may visit everything. “Disallow: /” means the reverse, the whole site is blocked.
People also search for this as: robots.txt generator, create robots.txt file, block AI bots robots.txt, robots.txt example, disallow all robots.txt, robots.txt for WordPress.
Sources and further reading
- Google Search Central: How to write and submit a robots.txt file
- Google Search Central: Introduction to robots.txt
- RFC 9309: Robots Exclusion Protocol
Formulas on this page are checked by automated tests against independent references. See how we test our tools.