Allowing robots to groups-only content

Posted: Sun Nov 26, 2006 6:42 am

I'm using a modification of static.php to serve my old static html through my RN 2.02.02 site. I've made a modification to secure certain directory structures to groups-only, but I want to continue allowing robots to index the content.

RN 2.00.20 had an areyouabot() in sessions.php, but this is deprecated in 2.02.02. What replaced it, and in which script will I find it? What, if any, problems are there around recognizing a robot request?

registered · Site Admin Joined: Jun 04, 2004 Posts: 6437

Very interesting question... I checked an old 2.02.00 version, but couldn't find any references to areyouabot. I also checked 1.05.00, 2.02.02 and the current 2.10.00 working code - nothing.

However, I did find that the areyouabot function was added as part of the GoogleTap (now TegoNuke Shortlinks) extension. It could be that the current version of ShortLinks uses a different approach / function. Maybe Montego can shed some light on that.

However, if you look at the Only registered users can see links on this board! Get registered or login!, you can find the code for this function, add it to your sessions.php, and use it where you like.

Posted: Sun Nov 26, 2006 12:05 pm

I have the function in a sessions.20002.php file. I assumed that it was in RN 2.00.02. I guess that I had added it with my original GT implementation, and took it out with a later change.

Thanks for the referral to Montego - that makes perfect sense.

registered · Posted: Sun Nov 26, 2006 3:45 pm

kguske is correct. That function was added originally in GT-NExtGEn. It might have even been there for the original GoogleTap, but that is before my time.

However, I think your question is much more complex than that. I guess you could use the principles in that GTNG function and incorporate it somehow, but you run the risk of upsetting potential new "customers". Even if you were successful at letting the search engines cache your "protected content", as soon as a new potential "customer" clicks on that link, they are immediately hit from your site with "you are not authorized to access this content". You also have the issue of the content being cached for a very long time on the search engines themselves, so if someone wants the content, why go to your site?

I certainly do not recommend it, but its your site.