How to make your robots.txt
- Pick a quick fill or type the paths every crawler should skip, one per line, starting with a slash. Leave both boxes empty to allow the whole site.
- Add the full address of each sitemap.
- Check the AI crawlers you want to block, or use one of the three AI buttons. Unchecked crawlers follow the same rules as everyone else.
- Copy or download the file and upload it to the root of your site, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
- Open that address in a browser to make sure the new file is the one being served.
The AI crawlers, by what they do
- AI training: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple), meta-externalagent (Meta) and CCBot (Common Crawl). Blocking these opts your pages out of training future models.
- AI search: OAI-SearchBot (ChatGPT search), Claude-SearchBot and PerplexityBot. Blocking these can keep your pages out of those assistants' answers and links.
- Fetches for a user: ChatGPT-User, Claude-User and Perplexity-User open a page when a person asks about it. OpenAI and Perplexity say robots.txt may not apply to theirs.
Blocking Google-Extended does not take you out of Google Search, AI Overviews or AI Mode. For those, see how to opt out of AI Overviews. To give AI tools a map of your site instead, see what goes in an llms.txt file.
Questions about robots.txt
What is a robots.txt file?
A robots.txt file is a plain text file at the root of a website, like example.com/robots.txt, that tells crawlers which pages they may visit. It must be named robots.txt, saved as UTF-8 text, and sit at the root of the host it applies to; a file in a subfolder is ignored.
Does robots.txt keep a page out of Google?
No. Google says robots.txt is not a way to keep a page out of its results: a blocked page can still be indexed, without a description, if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex tag, or put it behind a password.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls whether Google may use your pages to train Gemini models and to ground answers in Gemini. AI Overviews and AI Mode are part of Search and use Googlebot, so the controls for them are the nosnippet and max-snippet rules, not robots.txt.
Should I block AI crawlers?
It depends on what you want from AI tools. Blocking a search crawler such as OAI-SearchBot, Claude-SearchBot or PerplexityBot can keep your pages out of that assistant’s answers and links. Blocking a training crawler such as GPTBot or ClaudeBot opts your pages out of future model training without taking them out of AI search.
Do AI crawlers follow robots.txt?
OpenAI, Anthropic and Perplexity say their crawlers follow it. Their user-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person asked for the page.
What happens when a crawler is named in its own group?
A crawler follows only the one group that names it most specifically, and ignores the User-agent: * group. That is why this generator gives blocked AI crawlers a group of their own and leaves allowed ones to follow the rules for everyone.
Is my data sent anywhere?
No. The generator runs entirely in your browser. Nothing you type is sent to The Search Post or anyone else.