Robots.txt Generator and Tester

Create a correct robots.txt file in seconds, with presets for Blogger and WordPress, your sitemap, the folders you want to keep crawlers out of, and optional blocks for AI crawlers. Then test any URL against your rules to see whether a crawler is allowed to fetch it.

Block AI and other crawlers (optional)

Blocking these does not affect Google Search or Bing rankings. Google-Extended controls whether Google may use your content for its AI models (Gemini). It is not Googlebot.

Your robots.txt


  

Test a URL against these rules

The tester follows Google's documented rules: the most specific group for the crawler applies, the longest matching rule wins, * matches any characters, $ marks the end of the URL, and when an Allow and a Disallow rule are equally long, Allow wins.

What is a robots.txt file?

A robots.txt file is a plain text file at the root of your website, for example https://www.example.com/robots.txt. It tells search engine crawlers which parts of the site they may crawl. Well-behaved crawlers such as Googlebot and Bingbot read it before they crawl anything else. It is part of the Robots Exclusion Protocol, which became an official internet standard (RFC 9309) in 2022.

The file is made of groups. Each group starts with one or more User-agent lines naming the crawler, followed by Allow and Disallow rules for URL paths. You can also list your XML sitemaps with Sitemap lines, which helps search engines find your pages.

How to use this robots.txt generator

  1. Choose a preset. For most sites, Allow everything is the right choice. Search engines are good at deciding what to crawl.
  2. Enter your sitemap URL. On Blogger it is https://yourdomain.com/sitemap.xml. On WordPress with Yoast it is usually /sitemap_index.xml.
  3. Add only the paths you really want to keep crawlers out of, such as internal search results, admin areas or endless filter combinations.
  4. Optionally block AI training crawlers or third-party SEO bots.
  5. Test some important URLs with the tester, then copy or download the file.

Using a custom robots.txt on Blogger

Blogger creates a robots.txt for you automatically, and it already blocks /search pages and lists your sitemap. To change it, go to Settings › Crawlers and indexing, turn on Enable custom robots.txt, and paste the file from this tool. Keep the Mediapartners-Google group, which is the AdSense crawler. Leaving it unblocked lets AdSense read your pages and show relevant ads, and this matters when you apply for AdSense.

Common robots.txt mistakes

  • Blocking the whole site by accident. Disallow: / under User-agent: * stops all crawling. It is fine for a staging site, but a disaster on a live one.
  • Using robots.txt to hide pages from Google. A blocked URL can still be indexed if other sites link to it, but it is shown without a description. To keep a page out of search results, use a noindex meta tag and let Google crawl the page so it can see that tag.
  • Blocking CSS and JavaScript. Google renders pages like a browser. If it cannot load your styles and scripts, it may misjudge your layout and mobile-friendliness.
  • Forgetting that paths are case-sensitive. /Admin/ and /admin/ are different paths.
  • Treating robots.txt as security. The file is public, and bad bots ignore it. Protect private areas with a login.

Frequently asked questions

Does robots.txt affect rankings?

Not directly. It controls crawling, not ranking. Blocking important pages prevents Google from reading them, which can stop them from ranking well.

Should I block AI crawlers?

It is a business decision. Blocking GPTBot, CCBot or Google-Extended can stop your content from being used to train some AI models, and it does not change your normal search rankings. Some AI assistants use different user agents to fetch pages live, so check each company's documentation.

How do I check my live robots.txt?

Open yourdomain.com/robots.txt in a browser. Google Search Console shows the version Google last fetched in Settings › robots.txt.