GPTBot robots.txt rules: four snippets and what each one does
Block training, keep ChatGPT search, protect one folder, and apply the same opt-out to ClaudeBot and Google-Extended.
By ClaudeGPT Si Newsroom · Editorial team
· 2 min read
GPTBot robots.txt rules take two lines: a User-agent: GPTBot line and a Disallow or Allow line under it. Below are four snippets you can copy, what each one tells OpenAI, Anthropic and Perplexity, and how to confirm the result with the pay-per-run AI crawler checker on seogeoaeo.ai.
What GPTBot is
OpenAI describes GPTBot as the crawler that gathers content which may be used to train its generative AI foundation models. Disallowing it signals that your content should not go into training. It is not the bot behind ChatGPT search; that job belongs to OAI-SearchBot, which you control on its own line.
Snippet 1: keep GPTBot out of the whole site
User-agent: GPTBot
Disallow: /
This is the plain training opt-out. ChatGPT search is not affected, because OAI-SearchBot follows its own group.
Snippet 2: block training, keep ChatGPT search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
OpenAI says the two settings are independent, and this is the combination it uses as its own example. It suits a publisher who wants citations without feeding training data.
Snippet 3: block one folder only
User-agent: GPTBot
Disallow: /members/
Allow: /
Useful when part of the site is paid or private and the rest is public. Under the robots.txt standard, paths are matched from the start of the URL, and the longer, more specific rule wins when two match.
Snippet 4: the same opt-out for Anthropic, Perplexity and Google
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
ClaudeBot is Anthropic's training crawler; Anthropic asks you to repeat the rule on every subdomain you want to opt out. Google-Extended is a token, not a separate crawler, and controls Gemini training and grounding without touching Google Search. PerplexityBot is left out on purpose: Perplexity says it surfaces sites in search results and is not used to train foundation models, so blocking it costs citations for little gain.
The mistake that undoes all of this
A bot reads only the group that names it most specifically. Say your file has a long list of rules under User-agent: *, and you add a short User-agent: GPTBot group with one line. GPTBot now ignores every rule under * and follows that one line alone. The fix is to copy any rules GPTBot still needs into its own group.
Also, robots.txt is a polite request. Anthropic says its bots honor it, and OpenAI and Perplexity name robots.txt as the way to control theirs, but the file cannot force anything. If you need a hard block, that happens at your server or CDN. Anthropic notes that blocking its IPs can stop ClaudeBot from reading robots.txt at all, so the file is still the method it supports for an opt-out.
Confirm it worked
| Step | What to use |
|---|---|
| See which rule matches GPTBot on a given path | seogeoaeo.ai Robots.txt & AI Crawler Checker, 10 credits |
| See whether GPTBot still visits | seogeoaeo.ai AI Bot Log Analyzer on a pasted access log, 8 credits |
| Check the visit really came from OpenAI | Compare its IP with openai.com/gptbot.json |
Enter GPTBot as the crawler and the checker names the exact rule that applies to the path you test. For a two-line file you can read it yourself; the checker earns its keep once the file has several groups.