Can robots.txt tell the difference between a slash and a dash?

robots.txt fight

I’ve been working with my client, Gremlin Inc, to launch a content library covering the topic of Chaos Monkey. We have a PPC page that is blocked via robots.txt using a *very* similar URL structure to one of the subpages of this asset. The only difference is a subfolder – yet Google is blocking both pages from organic search. It appears like this may be a bug with the way Google processes URLs with robots.txt, but I wanted to open this up to the community and see if anyone has dealt with something similar.

Here’s the robots.txt:

Gremlin Robots.txt

Notice line 10:

Disallow: /chaos-monkey-simian-army/

The Simian Army subpage of the guide we’ve created, has a URL that’s supposed to be indexed:


But Google is blocking this result:

And we have to opt in for extra results just to get to it:

blocked results for gremlin's simian army page that should be indexed.

When i try to submit the page in search console, it tells me robots.txt is blocking the page:

When we test using the old Web Master Tools robots.txt tester – it says the URL should be allowed:

old WMT saying the URL is okay

So is Google allowing for a type of fuzzy match on near identical URLs when it comes to processing subfolders and dashes in URLs that are supposed to be blocked by robots.txt?


Please leave a comment if you’ve seen something similar.



We removed line 10 from robots.txt, resubmitted robots.txt using the old Web Master Tools, used fetch + render and submitted the page. We are back in, with a description:

result showing up properly

Definitely looks like a bug from where I am sitting…



Image via Bergen Offentlige Bibliotek

By jhtscherck

John-Henry Scherck is the owner of Growth Plays, a B2B content strategy and SEO consultancy based in Los Angeles. He works with founders, marketers, and investors to plan, build and refine growth marketing initiatives using a common sense approach.


  1. Interesting here is that 2 different Google tools are telling 2 different results. 1 saying it’s blocked by robots.txt and other says it’s allowed here it’s clear that 1 has bug …. I saw some kind of other bug in url inspection so I believe robots checker is correct

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.