Comments on: Does Google Index Your robots.txt?

By: piyush technoindian

piyush technoindian — Mon, 28 Nov 2011 08:04:31 +0000

i think URL removal request is the best way out of this problem and the link you posted about robots.txt of rishi lakahani’s website was amazing

By: Moosa Hemani

Moosa Hemani — Thu, 17 Nov 2011 17:03:30 +0000

Wow! Thank you for your comments people!

Peter as we discussed on twitter I assumed robots.txt is a normal page and submitted my URL removal request on Google and it worked out! I tried this on two different sites.

Bogdan, this is interesting! Thanks for the idea. I’ll look into it and will update, maybe your idea works better than mine…

Thanks to Tad for accepting my content and editing it for the readers!

By: Tadeusz Szewczyk

Tadeusz Szewczyk — Thu, 17 Nov 2011 11:26:37 +0000

No Peter, I just did the removal request in Google Webmaster Tools as advised by Moosa.

By: Peter Handley

Peter Handley — Thu, 17 Nov 2011 11:23:05 +0000

interesting test – was that with changing the header status of your robots.txt? I’d be loath to do that, as I want them to respect it, and if it were to 404 error, then I would have concerns that it wouldnt respect the items I am blocking in there.

When I get time (I’ve just got back from some holiday) I’ll test a removal request on the one that made me observe this

By: Tadeusz Szewczyk

Tadeusz Szewczyk — Thu, 17 Nov 2011 11:08:06 +0000

OK Bogdan, thank you for the feedback. On a sidenote: The test with the removal request of my own robots.txt has worked. It’s now de-indexed:

https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fseo2.0.onreact.com%2Frobots.txt

By: Bogdan

Bogdan — Thu, 17 Nov 2011 07:46:47 +0000

@tad
I don’t know exactly why it didn’t work. It is possible that eventually all my robots.txt files will be dropped out of the index. Might be a matter of time. I’m not sure.

Google says however that:
“by default pages are treated as crawlable, indexable, archivable, and their content is approved for use in snippets that show up in the search results, unless permission is specifically denied in a robots meta tag or X-Robots-Tag.”

By: Tadeusz Szewczyk

Tadeusz Szewczyk — Wed, 16 Nov 2011 14:09:13 +0000

Peter: Can’t you modify/manipulate the header status? Btw. I just submitted a removal request for my own robots.txt which is empty anyways.

Bogdan: Thank you for the feedback! Why do you think it didn’t work on some sites?

By: Bogdan

Bogdan — Tue, 15 Nov 2011 14:00:46 +0000

you can also try using the X-Robots-Tag HTTP header specifications with “noindex” as a response.
more info here:
http://code.google.com/web/controlcrawlindex/docs/robots_meta_tag.html

it’s easy to set it up via .htaccess. i’ve tried this on several sites. for some of them it worked, for others it didn’t…

By: Peter Handley

Peter Handley — Tue, 15 Nov 2011 13:44:04 +0000

Im not sure your second option, the removal request, would actually work (though I haven’t tested it). Presumably a robots.txt file would return a 200 header status, meaning that the removal request wouldn’t go through.

Therein lay my logic problem! I decided it wasn’t doing any harm and have left it, but it did find it curious! :)