Googlebot in Private Messages?

Involved Joined: Feb 08, 2005 Posts: 290 Location: USA

How do I block googlebot from Private Messages? I have the robot text set to not allow access to that module I'm pretty sure ... how is it that it's going there anyway?

registered · Site Admin Joined: Jun 04, 2004 Posts: 6437

How do you have robots.txt set to block it?

Posted: Sat Jan 13, 2007 10:58 am

If you are using the default robots.txt then Google is permitted in Private Messages in as much as, it is allowed to index legitimate links to it from within your site.
For example on your home page menu there are links to Private Message and also in the Memeberlist module if this is publicly accesible as well as forum pages.

As Private Messages cannot be sent from 'anonymous' you may as well ensure the Private Messages modules access permission is set to 'Registered Only' in nukes module administration area andhope Google will drop links already indexed.

Posted: Sat Jan 13, 2007 11:04 am

It will respect robots.txt if you have added a restriction against looking at the Private Messages module.

Posted: Sat Jan 13, 2007 3:38 pm

Mine looks like this ... not sure it's right mind you LOL

User-agent: Googlebot-Image
Disallow: /

User-agent: *
Crawl-delay: 20
Disallow: /modules.php?name=Top&querylang=union%20select%200,pwd?id=honeytrap
Disallow: /_private/
Disallow: /_vti_bin/
Disallow: /_vti_cnf/
Disallow: /_vti_log/
Disallow: /_vti_pvt/
Disallow: /_vti_txt/
Disallow: /abuse/
Disallow: /admin/
Disallow: /blocks/
Disallow: /cgi-bin/
Disallow: /db/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /themes/
Disallow: /admin.php
Disallow: /config.php
Disallow: /demohack.php
Disallow: /cplogin.php
Disallow: /hackattempt.php
Disallow: /login.php

Doesn't Disallow: /modules block the bot? If not what should my robots.txt look like please? Thanks : o}

Posted: Sat Jan 13, 2007 3:50 pm

I've also seen the bot in Your Account and Profiles, I need to block it from those as well!

Posted: Sat Jan 13, 2007 4:22 pm

PMs are set to registered only. But Your Account has to be open to all. Profiles, I assume, is another way of saying Your Account? Anyway, I don't want the bots into anything *private*. My Member List is Admin only now, so I'd rather the bots weren't into that either (to keep people's emails private).

How do I add on to the robots.txt to keep them out of those three areas?

Posted: Sat Jan 13, 2007 4:51 pm

Once you set them to registered, or admin only Googlebot should not be able to access them, so you should be safe.

Posted: Sat Jan 13, 2007 5:33 pm

It is accessing them, and it is Registered only ... that's why I'm posting LOL. I don't want the bot in any of the private stuff like Member List, Private Messages, Your Account or Profiles. The only one of those that is public is Your Account, and it has to be so people can join. So, how to keep the bot out of those private areas?

Or, are you saying even though they are getting into those areas they are not logging what they find? If that's the case why are they in them at all then?

Posted: Sat Jan 13, 2007 8:48 pm

What links is Google getting to exactly?
I am assuming you have some data which tells you were google is going so it might help if you can give a couple of example links.

Posted: Sat Jan 13, 2007 8:49 pm

/modules will not block bots. But

Code:

Disallow: /modules.php?name=Private_Messages

should.

Posted: Sat Jan 13, 2007 9:47 pm

Where should the robots.txt file be btw? I currently have two, on in the domain root, and one in the nuke root ... do I need two? Which, if only one necessary, should I keep?

Posted: Sat Jan 13, 2007 9:51 pm

Googlebot samples from IP Tracking ...

/nukefolder/modules.php?name=Private_Messages&file=index&mode=post&u=56 2007-01-13 03:44:10
/nukefolder/modules.php?name=Your_Account&redirect=posting&mode=quote&p=681

registered · Posted: Sat Jan 13, 2007 11:55 pm

Mm that's interesting. Can we get the IPs on those? It may not really be Google bot, esp if there are no public links on your site to those

Posted: Sun Jan 14, 2007 1:21 am

This is only a small list of where this one has been but the Your Account and Private Messages concern me ...

66.249.72.237 crawl-66-249-72-237.googlebot.com
/nukefolder/modules.php?name=Your_Account&redirect=posting&mode=quote&p=639 2007-01-11 10:48:23
/nukefolder/modules.php?name=Your_Account&redirect=posting&mode=quote&p=638 2007-01-11 10:43:20
/nukefolder/modules.php?name=Your_Account&redirect=posting&mode=quote&p=645 2007-01-11 10:38:21
/nukefolder/modules.php?name=Your_Account&redirect=posting&mode=quote&p=641 2007-01-11 10:23:21
/nukefolder/modules.php?name=Private_Messages&file=index&mode=post&u=53

Here are two more IPs from the same domain
66.249.66.163 crawl-66-249-66-163.googlebot.com
66.249.66.40 crawl-66-249-66-40.googlebot.com

A Whois search brings up google for these IPs.

Posted: Mon Jan 15, 2007 2:45 am

Well the first ones are valid redirects, because Google sees links such as
modules.php ? name=Forums&file=posting&mode=quote&p=639
It also sees the second links.

These are public URLs that phpBB gives out. They do not indicate that Google bot is actually looking at any private messages or is trying to post something

Posted: Thu Jan 18, 2007 12:19 am

OK so not to be concerned about it showing as Private Messages or Your Account? Or should I Disallow from robot.txt for both of these features just to be safe? Thanks for you help btw!

Posted: Thu Jan 18, 2007 12:48 am

You could disallow them. It really doesn't matter, I don't think Google will index them because there is no original content on them