Create a correct robots.txt file in seconds, with presets for Blogger and WordPress, your sitemap, the folders you want to keep crawlers out of, and optional blocks for AI crawlers. Then test any URL against your rules to see whether a crawler is allowed to fetch it.
Your robots.txt
Test a URL against these rules
The tester follows Google's documented rules: the most specific group for the crawler applies, the longest matching rule wins, * matches any characters, $ marks the end of the URL, and when an Allow and a Disallow rule are equally long, Allow wins.
What is a robots.txt file?
A robots.txt file is a plain text file at the root of your website, for example https://www.example.com/robots.txt. It tells search engine crawlers which parts of the site they may crawl. Well-behaved crawlers such as Googlebot and Bingbot read it before they crawl anything else. It is part of the Robots Exclusion Protocol, which became an official internet standard (RFC 9309) in 2022.
The file is made of groups. Each group starts with one or more User-agent lines naming the crawler, followed by Allow and Disallow rules for URL paths. You can also list your XML sitemaps with Sitemap lines, which helps search engines find your pages.
How to use this robots.txt generator
- Choose a preset. For most sites, Allow everything is the right choice. Search engines are good at deciding what to crawl.
- Enter your sitemap URL. On Blogger it is
https://yourdomain.com/sitemap.xml. On WordPress with Yoast it is usually/sitemap_index.xml. - Add only the paths you really want to keep crawlers out of, such as internal search results, admin areas or endless filter combinations.
- Optionally block AI training crawlers or third-party SEO bots.
- Test some important URLs with the tester, then copy or download the file.
Using a custom robots.txt on Blogger
Blogger creates a robots.txt for you automatically, and it already blocks /search pages and lists your sitemap. To change it, go to Settings › Crawlers and indexing, turn on Enable custom robots.txt, and paste the file from this tool. Keep the Mediapartners-Google group, which is the AdSense crawler. Leaving it unblocked lets AdSense read your pages and show relevant ads, and this matters when you apply for AdSense.
Common robots.txt mistakes
- Blocking the whole site by accident.
Disallow: /underUser-agent: *stops all crawling. It is fine for a staging site, but a disaster on a live one. - Using robots.txt to hide pages from Google. A blocked URL can still be indexed if other sites link to it, but it is shown without a description. To keep a page out of search results, use a
noindexmeta tag and let Google crawl the page so it can see that tag. - Blocking CSS and JavaScript. Google renders pages like a browser. If it cannot load your styles and scripts, it may misjudge your layout and mobile-friendliness.
- Forgetting that paths are case-sensitive.
/Admin/and/admin/are different paths. - Treating robots.txt as security. The file is public, and bad bots ignore it. Protect private areas with a login.
Frequently asked questions
Does robots.txt affect rankings?
Not directly. It controls crawling, not ranking. Blocking important pages prevents Google from reading them, which can stop them from ranking well.
Should I block AI crawlers?
It is a business decision. Blocking GPTBot, CCBot or Google-Extended can stop your content from being used to train some AI models, and it does not change your normal search rankings. Some AI assistants use different user agents to fetch pages live, so check each company's documentation.
How do I check my live robots.txt?
Open yourdomain.com/robots.txt in a browser. Google Search Console shows the version Google last fetched in Settings › robots.txt.