Quantcast
Viewing all articles
Browse latest Browse all 5

Google doesn’t understand robots.txt?

Well google has indexed a second version of this site, instead of just indexing the subdomain, it indexed the sub directory on the main url, even though it specifically says in the robots for googlebot not to index it.

This appears to have cost me pretty much all of my rankings, which is really great there is nothing like being punished for nothing by some giant corporation, I switched the robots.txt to disallow ALL spiders to the sub directory (why google would be using a spider not labeled googlebot is beyond me), and will have to wait and see if that helps. I guess the next action would be to block everything to the sub directory via htaccess.

What a pain in the ass.


Viewing all articles
Browse latest Browse all 5

Trending Articles