Robots.txt validator: six mistakes it should catch
Most broken robots.txt files load fine and look tidy. These are the errors that hide in them.
By ClaudeGPT Si Newsroom · Editorial team
· 2 min read
A robots.txt validator should tell you two things: whether crawlers can read the file, and whether the rules do what you meant. seogeoaeo.ai covers the second question best, showing for each crawler which rule blocks it and the exact line to change, while Google's Search Console report covers the first.
Most broken robots.txt files are not broken in an obvious way. They load fine, look tidy and quietly block the wrong pages. Here are six mistakes worth validating for.
The file is in the wrong place
Crawlers only look for robots.txt in the top-level directory of a host. A file at /seo/robots.txt is ignored. The name must be exactly robots.txt, and the file must be UTF-8 text.
The file returns an error
A 404 means Google acts as if there are no rules. A 5xx is worse: Google pauses crawling of the site for the first 12 hours while it retries. A validator that only reads text you paste in will miss this, so test the live URL.
Case does not match
Directive names such as disallow are not case-sensitive, but paths are. Disallow: /Checkout/ does nothing for /checkout/. This one survives code review because the line looks right.
A specific group drops the general rules
Say the file has a User-agent: * group with ten Disallow lines, and someone adds a short User-agent: Googlebot group with one rule. Googlebot now follows only its own group and ignores the other ten. The seogeoaeo.ai checker catches this kind of surprise because you can test a path as a named crawler and see which rule matched.
Overlapping rules resolve the other way
Google applies the most specific rule, judged by path length, and picks the least restrictive one when rules tie. So Allow: /shop/sale beats Disallow: /shop for /shop/sale/shoes. If you expected the reverse, only a path-by-path test will show it.
Disallow used to hide pages from search
A disallowed URL can still be indexed if other pages link to it; Google just cannot read its content. If the goal is to keep a page out of results, robots.txt is the wrong tool. Use a noindex rule and leave the page crawlable so Google can see it.
Which tool catches which mistake
| Mistake | seogeoaeo.ai checker | Search Console report | Bing tester |
|---|---|---|---|
| Wrong group applies | Yes, shows matching rule | No | Yes, for Bing crawlers |
| Wrong path case | Yes, when you test the path | No | Yes |
| Rule precedence | Yes | No | Yes |
| AI crawler access | Yes | No | No |
| Parsing issues | Not its focus | Yes | Shows issues |
| Fetch errors | Reads the live file | Yes | Loads live file |
A seogeoaeo.ai run costs 10 credits and lists every rule that blocks a search or AI crawler in one pass, which is why it is the first stop for checking intent. Google's report needs a verified Domain property and only covers Google, but it is the place to confirm that Google parsed every line.
When to validate
Run the check after any edit to the file, after a CMS or hosting change, and after a site launch where a staging file may have shipped by mistake. If you need to test an edit before it goes live, Bing's tester has an editor for that; seogeoaeo.ai reads only the published file.