How to use the Robots.txt Generator
- 1. Set the rulesChoose allow or block by default, then list paths to block and any exceptions.
- 2. Add extrasOpt out of AI crawlers, block other bots, set a crawl delay and add your sitemap.
- 3. Test and downloadCheck a path against the rules, then download robots.txt for your site's root.
How robots.txt rules work
A robots.txt file sits at the root of a domain (/robots.txt) and is split into groups that start with User-agent. A crawler uses the group that names it most specifically, or the * group if none does. Inside a group, the longest matching rule wins, and when an Allow and a Disallow rule are the same length, Allow wins. * matches any characters and $ marks the end of the URL, so /*.pdf$ blocks every PDF. The tester follows these rules from the robots.txt standard (RFC 9309). Everything runs in your browser; nothing you type or choose is uploaded.
robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it, just without a description. To keep a page out of Google, let it be crawled and add a noindex meta tag instead.
Worked example
The defaults allow crawling but block /admin/, /cart/, /checkout/ and sorted listing URLs (/*?sort=), with an exception for /admin/help/ and a sitemap at example.com. The file has 7 directives in 1 user-agent group. Testing /checkout/step-1 shows it is blocked for Googlebot and Bingbot by Disallow: /checkout/. /admin/help/faq would be allowed, because Allow: /admin/help/ is the longer match.
About AI crawlers
- The AI option adds a block for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent and PerplexityBot. Each company documents its own crawler name, and names change, so review the list from time to time.
- Blocking Google-Extended doesn't remove you from Google Search; Googlebot is a separate crawler.
- robots.txt is a request, not a lock. Crawlers that ignore it need to be blocked at the server or firewall.
Frequently asked questions
Where do I put the file?
At the root of each domain or subdomain, so it opens at https://www.yoursite.com/robots.txt. A file in a subfolder is ignored.
Does Google follow Crawl-delay?
No. Google ignores it; Bing and some other crawlers respect it.
Is the robots.txt generator free?
Yes. You can open the robots.txt generator and fill in every option without signing up. When you generate the file for the first time, we ask once for your name and WhatsApp number; after that it, and every other Uncodemy tool, opens straight away.