Robots.txt Guide: Rules, Examples and Mistakes to Avoid

By ·

A robots.txt file is a small plain-text file that tells search engine crawlers which parts of your site they may crawl. One wrong line can stop Google from crawling your whole site, and one misunderstanding about what it does can leave private pages visible in search. It is worth ten minutes to get right.

This guide explains how robots.txt works, the rules Google supports, how to read and write them, copy-ready examples for common situations including Blogger and WordPress, and the mistakes that cause the most trouble.

What robots.txt does, and what it does not do

Robots.txt controls crawling: whether a well-behaved bot requests a URL. Google's introduction to robots.txt describes its main purpose as managing crawler traffic to your site, for example to avoid wasting crawl activity on unimportant or similar pages.

It does not reliably control indexing. If a blocked page is linked from elsewhere, Google can still index its URL without crawling it, usually showing it with no description. To keep a page out of search results, use a noindex meta tag or HTTP header instead, and do not block that page in robots.txt, because Google has to crawl the page to see the noindex. Google explains this in its guide to blocking indexing with noindex.

Robots.txt is also not a security feature. The file is public, anyone can read it, and badly behaved bots can ignore it. Never use it to "hide" private areas; protect them with a login.

Where the file lives

  • It must be named exactly robots.txt, in lowercase.
  • It must sit at the root of the host: https://www.example.com/robots.txt. A file in a subfolder is ignored.
  • Each host and protocol has its own file. https://blog.example.com needs its own robots.txt, separate from https://www.example.com.
  • It must be UTF-8 plain text. Google's robots.txt specification notes that Google reads up to 500 KiB of the file and ignores anything beyond that, which is far more than a typical site needs.

The rules Google supports

RuleWhat it doesExample
User-agentNames the crawler the following group of rules applies to. * means all crawlers that do not have a more specific group.User-agent: Googlebot
DisallowA path the crawler should not crawl. An empty value means nothing is disallowed.Disallow: /search
AllowA path that may be crawled, used to make an exception inside a disallowed folder.Allow: /search/label/
SitemapThe full URL of an XML sitemap. Can appear anywhere in the file, and you can list several.Sitemap: https://www.example.com/sitemap.xml

Google ignores some rules other crawlers recognize, such as crawl-delay. Bing, by contrast, does honor crawl-delay. If you need to limit Googlebot's crawl rate, robots.txt is not the tool for it.

Wildcards

  • * matches any sequence of characters. Disallow: /*?sort= blocks any URL that contains ?sort=.
  • $ marks the end of the URL. Disallow: /*.pdf$ blocks URLs ending in .pdf.

How conflicts are resolved

Paths are case-sensitive and match from the start of the URL path. When both an Allow and a Disallow rule match a URL, Google uses the most specific rule, meaning the one with the longest matching path. If they are equally long, Google uses the least restrictive one, which is Allow.

A crawler follows only the single group that best matches its name. If you create a User-agent: Googlebot group, Googlebot ignores the User-agent: * group completely, so repeat any rules you want it to follow.

How to create a robots.txt file step by step

  1. List what should not be crawled. Typical candidates are internal search results, cart and checkout pages, filtered or sorted duplicates of category pages, and admin areas. If you cannot name a reason to block something, do not block it.
  2. Check that nothing important is caught. Never block CSS, JavaScript or image folders your pages need, because Google renders pages to understand them.
  3. Write the rules. Start from a template below, or build the file with the robots.txt generator and tester, which has Blogger and WordPress presets.
  4. Test specific URLs against your draft. The tester in our tool shows which rule matches a URL and whether it is allowed or blocked.
  5. Upload or paste the file at the root of your site.
  6. Confirm in Search Console. The robots.txt report (under Settings) shows the version Google last fetched and any parsing problems.

Copy-ready examples

Allow everything

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

An empty Disallow means nothing is blocked. This is a perfectly good robots.txt for many small sites.

Block internal search results

User-agent: *
Disallow: /search
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Blogger default (with AdSense)

Blogger generates a robots.txt for every blog. It looks like this:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Mediapartners-Google is the AdSense crawler, which reads pages to show relevant ads. The empty Disallow gives it full access. The /search rule keeps label and search result pages from being crawled; remove it only if you deliberately want label pages crawled. You can replace the default under Settings > Crawlers and indexing > Custom robots.txt, but if the default does what you need, leave it alone.

Typical WordPress

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap_index.xml

Block sorting and filter parameters

User-agent: *
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?filter=

Opt out of some AI training crawlers

Several AI companies publish crawler names that respect robots.txt. For example, GPTBot (OpenAI), CCBot (Common Crawl) and Google-Extended, a token Google provides to control whether content is used for its AI models without affecting Google Search.

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Check each company's current documentation, as crawler names change over time.

Common mistakes

  • Disallow: / left from a staging site. This single line blocks the entire site for the matching crawlers. It is the first thing to check if a site suddenly loses traffic after a launch.
  • Blocking pages to remove them from Google. Use noindex instead, and keep the page crawlable until it drops out.
  • Using a noindex line inside robots.txt. Google does not support it.
  • Blocking CSS or JS files that pages need to render properly.
  • Wrong case. Disallow: /Private does not block /private.
  • Blocking the AdSense crawler on a monetized site, which can lead to less relevant ads or none.

FAQ

Do I need a robots.txt file at all?

No. If there is no file, crawlers assume they may crawl everything. Many small sites are fine with a simple file that just lists the sitemap.

How quickly does Google see changes?

Google generally caches robots.txt for up to 24 hours, so changes are not instant. You can ask for a recrawl from the robots.txt report in Search Console.

Does robots.txt stop other people from seeing my pages?

No. It is a request to crawlers, not access control. Anyone can still visit a URL directly.

Conclusion

A good robots.txt is usually short. Block only what you have a clear reason to block, never block what pages need to render, use noindex rather than robots.txt to keep pages out of search, and always test URLs before you save changes. When in doubt, a file that allows everything and lists your sitemap is a safe starting point. Next, make sure the sitemap it points to is accurate, which is easy with the XML sitemap generator.

Want keyword ideas for your own topic? Use the free keyword research tool → or browse all free SEO tools.

Comments