Three small pieces of text decide a lot about how a site appears in search: the robots.txt file, the robots meta tag, and the title and description. They are often confused with one another, and mixing them up is one of the most common ways to make a page disappear from Google by accident. This guide explains what each one controls and how to set them up properly.
Crawling, indexing and display are different things
- Crawling is a search engine visiting a page. robots.txt controls this.
- Indexing is a search engine keeping the page so it can show it in results. The noindex tag controls this.
- Display is how the result looks. The title, description and Open Graph tags influence this.
The most important consequence: if you block a page in robots.txt, search engines cannot read it, so they cannot see a noindex tag on it either. A blocked page can still appear in results as a bare address if other sites link to it. To keep a page out of search, allow it to be crawled and add a noindex tag.
robots.txt in five minutes
robots.txt is a plain text file at the top level of your site, so it opens at yourdomain.com/robots.txt. It is a request that well-behaved crawlers follow. It is not a lock, and anyone can read it, so never use it to hide private pages. A typical file looks like this:
Disallow: /admin/
Disallow: /cart/
Sitemap: https://yourdomain.com/sitemap.xml
| Line | What it does |
|---|---|
| User-agent | Names the crawler the rules apply to. * means all of them. |
| Disallow | A path not to crawl. An empty Disallow means nothing is blocked, and Disallow: / blocks the whole site. |
| Allow | An exception inside a blocked path. |
| Sitemap | The full address of your XML sitemap. |
| Crawl-delay | Asks for a pause between requests. Google ignores it. |
Google also understands * as a wildcard and $ to mark the end of an address, for example Disallow: /*.pdf$.
A few practical details: the file must be named robots.txt in lowercase, saved as plain UTF-8 text, and served from the root of each domain. Subdomains need their own file. Google reads only the first 500 KiB, and it may keep a copy for up to a day, so changes are not instant. The robots.txt generator writes the file for you and warns about common mistakes.
Keeping a page out of search: noindex
For pages such as thank-you pages, internal search results or draft content, add <meta name="robots" content="noindex"> to the page head, and make sure the page is not blocked in robots.txt. The meta tag generator can write the tag for you.
Title and description
The title is the clickable headline in the result. Write a clear, specific title for every page, put the main topic first, and keep it to about 50 to 60 characters. Google cuts titles by pixel width, not by character count, so a title full of wide letters can be trimmed sooner than one full of narrow ones. The snippet preview measures the real width. The meta description is the summary under the title. It does not directly change rankings, but a clear one earns more clicks. Around 70 to 160 characters is a good range. Google often rewrites both when it thinks another piece of your page matches a search better, so writing accurate text is the best way to keep yours.
Canonical URLs
If the same page can be reached at several addresses, for example with and without tracking parameters, a canonical link tells search engines which address you prefer. Google treats it as a strong hint. Every page should point to its own preferred address, with the full https:// URL.
What about AI crawlers?
Several companies run crawlers that collect web pages to train AI models, each with its own robots.txt name, such as GPTBot, CCBot, ClaudeBot and PerplexityBot. You can ask them not to visit by adding a group for each one with Disallow: /. Only crawlers that respect robots.txt will obey it, and it does not remove anything already collected. Google-Extended is a separate token that controls whether content is used for some of Google’s AI products. Blocking it does not affect your normal Google Search results. Crawler names change, so check each company’s documentation for the current one.
A checklist
- Open yourdomain.com/robots.txt in a browser and read it.
- Make sure it does not block pages, CSS or JavaScript you want to rank.
- Give every page its own title and description.
- Use noindex, not robots.txt, to keep a page out of results.
- Check your pages with the URL Inspection tool and the robots.txt report in Google Search Console.
Sources and further reading
- Google Search Central: How to write and submit a robots.txt file
- Google Search Central: Introduction to robots.txt
- Google Search Central: Influencing your title links in search results
- Google Search Central: Control your snippets in search results
- Google Search Central: What is URL canonicalization
- RFC 9309: Robots Exclusion Protocol
Numbers in worked examples are calculated by code and checked by automated tests. How we test.