If you have searched for the best custom robots.txt for Blogger, you have probably seen two camps: leave the default file alone, or paste a longer "perfect" file that blocks date archives with a /20* rule. The second option is popular, and it is also the one most likely to block your actual posts if a single line is missing.
A good custom robots.txt for Blogger stays close to the file Blogger already generates, keeps the AdSense crawler unblocked, lists both of your sitemaps, and leaves archive indexing to robots meta tags instead of crawl blocks. This guide gives you a copy-and-paste file, shows how to enable it in Settings, and explains the rules worth skipping.
The best custom robots.txt for Blogger
Blogger already creates a robots.txt for every blog. At the time of writing, the official Blogger blog itself uses a file like this (with its own domain in the sitemap line):
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /share-widget
Allow: /
Sitemap: https://www.example.com/sitemap.xml
That is a solid baseline. The one useful addition for most blogs is Blogger's pages sitemap, which lists static pages such as About or a tools page. A safe custom file looks like this. Replace the domain with yours, including www if you use it:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /share-widget
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-pages.xml
If you are still on a Blogspot address, use that host in the sitemap lines, for example https://yourblog.blogspot.com/sitemap.xml.
This file does three jobs and no more:
- Gives
Mediapartners-Google, the crawler Google uses for AdSense and related ad products, an emptyDisallow, which means it may crawl everything. Google's documentation on special-case crawlers notes that this crawler ignores the*group, so it needs its own rules. - Blocks
/search, which covers Blogger's built-in search results and label URLs that begin with/search/label/, and blocks/share-widget, which is part of Blogger's current default file. - Points crawlers to both the posts sitemap and the pages sitemap.
If this already matches what you need, you may not need a custom file at all. Only switch one on when you have a reason, such as adding the pages sitemap or a clearly understood extra rule.
How to enable custom robots.txt in Blogger
Blogger's help page on managing your blog's settings explains that a custom robots.txt replaces the default file. You are not appending a few lines; you are supplying the whole file, so keep the AdSense group and the sitemap lines.
- In the Blogger dashboard, open Settings and scroll to Crawlers and indexing.
- Turn on Enable custom robots.txt.
- Paste the file above, with your real domain in both sitemap lines.
- Click Save.
- Open
https://www.yourdomain.com/robots.txtin a browser and check that the saved text is exactly what you pasted. - In Google Search Console, open the robots.txt report (under Settings) and confirm Google can fetch the new file. You can also test individual URLs with the robots.txt generator and tester, which includes a Blogger preset.
Rules to avoid in a Blogger robots.txt
Do not block posts with Disallow: /20*
Many "perfect robots.txt" tutorials add Disallow: /20* to stop crawlers from reaching monthly archive pages, then add Allow: /*.html so posts still get through. Blogger post URLs look like /2026/11/your-slug.html, so they start with /20 as well. If you forget the Allow line, or write it incorrectly, you can block every post from the year 2000 onward.
Even when both lines are present, the pattern is brittle. Google's robots.txt specification uses the most specific matching rule, and equally specific conflicts are resolved in favour of the least restrictive rule. That is a lot of room for a typo. A safer way to keep thin archive and search pages out of results is a noindex robots meta tag, which you can set under Custom robots header tags. Our guide to custom robots header tags in Blogger covers a safe configuration.
Google's introduction to robots.txt is also clear on a related point: robots.txt controls crawling, not indexing. To keep a page out of search results, use noindex or password protection, and do not block the page in robots.txt if you want Google to see that noindex.
Do not block ?m=1
Blogger appends ?m=1 when it serves the mobile version of a page. Blocking those URLs in robots.txt can stop Google from crawling the mobile version. Leave them alone, and rely on the canonical tag instead. If those URLs appear in Search Console, check that each page still points to the clean URL as its canonical version.
Do not remove the AdSense group
If you run AdSense or think you might apply, keep the Mediapartners-Google group with an empty Disallow. Removing it, or putting a site-wide Disallow: / in a group that this crawler follows, can affect how ads are matched to your pages.
Do not use robots.txt as a lock
The file is public, and poorly behaved bots can ignore it. Never list private folders there as a way to hide them.
Optional: blocking some AI training crawlers
If you want to ask certain AI training crawlers not to use your content, you can add separate groups for the crawlers those companies document. This is optional, and it is separate from Google Search. For example, OpenAI's crawler documentation describes GPTBot as the bot used for training generative models, and OAI-SearchBot as the bot used for ChatGPT search. Blocking one does not automatically block the other.
User-agent: GPTBot
Disallow: /
Add groups like this below your main rules, and re-check each company's current documentation before you rely on a name. For the rest of the syntax, see our robots.txt guide.
How to check that nothing important is blocked
- Visit
/robots.txtand read it as a crawler would: everyDisallowpath is a prefix, and paths are case-sensitive. - Test a homepage URL, a post URL, a static page URL such as
/p/about.html, a label URL and a search URL in the robots.txt tester. - Confirm both sitemap URLs load. The posts sitemap is
/sitemap.xml. The pages sitemap is/sitemap-pages.xml. - In Search Console, watch the robots.txt report and the Page indexing report for sudden spikes in "Blocked by robots.txt" on URLs you want crawled.
FAQ
Is there one official "best" custom robots.txt for Blogger?
No. Blogger's own default is already a sensible file. The safest custom version is that default, plus a pages sitemap line, with no extra blocks unless you can explain exactly which URLs they affect.
Why is my custom robots.txt not working?
Check that you saved the setting, that you are looking at the same host you edited (www versus non-www, or Blogspot versus a custom domain), and that you included a complete file rather than a fragment. Then wait: Google can cache robots.txt for up to a day.
Should I enable custom robots.txt if the default is fine?
No. Turn it on when you need to change the default, such as adding sitemap-pages.xml. If you enable it and leave the box empty, you can replace a working file with nothing useful.
Does a longer robots.txt help SEO?
No. Extra Disallow lines are not an advantage. They only help when they stop crawlers from wasting time on URLs you have a clear reason to exclude.
Should I block Blogger date archives in robots.txt?
Usually no. If you do not want archive pages in search results, noindex them with custom robots header tags and leave them crawlable.
Conclusion
The best custom robots.txt for Blogger is a short, boring file: allow the AdSense crawler, block Blogger search and share-widget URLs, allow everything else, and list both sitemaps. Skip the clever archive blocks, test a few real URLs after you save, and use header tags when your goal is indexing rather than crawling. If you want a starting file you can edit and test first, use the Blogger preset in the robots.txt generator.
Comments
Post a Comment