robots meta tag suggestion

Regular Joined: Jun 09, 2004 Posts: 79

I really appreciate being able to override the meta tags, but I wish I could do this for the robots tag. I would like to be able to keep some pages out of the search engines - GCalendar, Feedback, Legal, Privacy, Web_Links, etc. Unless I'm mistaken, it seems I now have to allow all or nothing. I've noticed Google uses a lot of bandwidth searching the calendar and links, and I see no reason to have them in search engine results.

Posted: Wed Jan 12, 2011 4:07 pm

Somehow you should be able to do that in the robots.txt.
But hornestly i dont now how to do that exactly.
However for me it seem you not need to do code manipulation, if i may assume that is what you might assumed.

registered · Posted: Wed Jan 12, 2011 5:15 pm

Google is supposed to follow the disallow in your robots.txt. You can disallow those modules you don't want indexed. You can wild card those module links and get all the secondary links too.

Posted: Wed Jan 12, 2011 11:44 pm

Disallowing in robots.txt doesn't work. The pages still show "index, follow" and are getting indexed. I'd like to be able to change some, but not all, modules to "no index, no follow".

Posted: Thu Jan 13, 2011 6:31 am

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

Posted: Thu Jan 13, 2011 8:03 pm

Thanks for the link, nuken; however, it doesn't tell me anything new. The exclusions are already in robots.txt but Google and Yahoo still crawl and index. I feel this is because when you look at the source code of the excluded page, it still says "index, follow". That's the reason I made the suggestion to add the robots meta tag to the override feature.

I had this issue before with Yahoo, and information on their site specifically states you must use the robots meta tag to keep a page excluded. Using robots.txt is insufficient.