Robots.txt Generator

How robots.txt works

robots.txt is a plain text file at the root of your site that tells crawlers which URLs they may fetch. The rules were standardised in 2022 as RFC 9309. The file is made of groups. Each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules:

User-agent: GPTBot
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Four rules decide what a crawler does:

1. It picks its own group. A crawler uses the group that names it (matching ignores upper and lower case). Only if no group names it does it use the * group. A bot with its own group ignores the * rules entirely, so the generator repeats your path rules for any AI bot you explicitly allow.
2. The longest matching rule wins, wherever it appears in the group. Above, /admin/help/page is allowed because Allow: /admin/help/ (12 characters) is longer than Disallow: /admin/ (7).
3. Allow wins a tie between equally long rules.
4. * matches any characters and $ marks the end. Disallow: /*.pdf$ blocks every URL ending in .pdf, but not /file.pdf?v=2.

Sitemap lines are not part of RFC 9309, but every major search engine reads them. They take a full URL and apply to the whole file. Crawl-delay is non-standard: Google, Apple and Amazon ignore it, while Bing and Anthropic honour it.

The AI crawlers, by purpose

Not every AI bot does the same job, and blocking the wrong one can cost you traffic. We checked each operator’s documentation on 22 October 2026:

Two tokens are special. Google-Extended and Applebot-Extended are not crawlers at all. Google and Apple keep crawling with Googlebot and Applebot, and these tokens only decide whether the content may be used for their AI models. Blocking them does not affect Google Search or Siri.

Some operators say their user-requested fetchers may not apply robots.txt, because a person asked for the page. Bytespider has no official documentation, and reports disagree on whether it obeys the file. We mark such entries in the list.

Worked examples

A blog that wants AI citations but no AI training. Choose Block AI training. Training crawlers get Disallow: /, while OAI-SearchBot, Claude-SearchBot and PerplexityBot follow your normal rules, so AI search can still quote and link to you. In the tester, GPTBot on /post/hello shows Blocked and OAI-SearchBot shows Allowed.

A staging site. Tick Block the whole site. Every crawler gets Disallow: /. Remember that robots.txt doesn’t hide pages from people; protect staging with a password too.

A shop with filter URLs. Block /cart, /checkout/ and /*?sort= so crawlers don’t waste time on endless filter combinations. Then test /shoes?sort=price (blocked) and /shoes (allowed).

Validating an existing file

Paste any robots.txt into Validate & test. The checker explains every line in plain English and flags common mistakes:

• rules placed before any User-agent line, which no crawler will apply;
• typos such as Disalow, and unknown fields;
• paths that don’t start with /, and sitemap entries that aren’t full URLs;
• Noindex: lines, which Google stopped supporting in 2019;
• user-agent values that are full browser strings instead of a bot token;
• files over 500 KiB, the most Google reads.

Tips

• Robots.txt is public. Anyone can read it, so don’t list secret paths in it. Use authentication for private areas.
• Blocking crawling is not blocking indexing. A blocked URL can still appear in Google results if other sites link to it. Use a noindex meta tag, which crawlers can only see if they’re allowed to fetch the page.
• Changes are not instant. Crawlers cache robots.txt, typically for up to 24 hours.
• Tell AI tools what to read. Alongside robots.txt, an llms.txt file points AI assistants to your most useful pages. Clean Markdown versions of those pages, made with document to Markdown, work best.