Google Page ranking

Posted: Sun Jan 16, 2005 11:10 pm

how can i tell if googlebots are crawling my site? or if i've been banned for some odd reason? My site has been live for quite a while now and i don't have a page ranking.

any ideas? Thanks.

registered · Posted: Mon Jan 17, 2005 1:53 am

candy wrote:

how can i tell if googlebots are crawling my site? or if i've been banned for some odd reason? My site has been live for quite a while now and i don't have a page ranking.

any ideas? Thanks.

The #1 way I track bots is by running a block called 'Who-is-Where' on my site. When you're logged on as an admin, instead of seeing 'Visitor' or 'Member' in the 'Who-is-Where' block, you'll see the IP addy. After a while, bots stick out like a sore thumb...

The #2 way is to simply look in my MSA stats. That's a no-brainer, with the bots listed by name.

Having said that, bots are kinda dumb, and they will occasionally try to run sql injections against my site, and so forth. When they do that, they get automatically banned. So, from time-to-time I check my ban list and see if any of those little boogers are listed. If they are, I remove them from my ban list and make sure they are excluded from 'auto-banning' in the future (1,921 IP's and counting!).

Banning bots is a 'good' way to get your site removed from a search engine. MSN bots are particularily sensitive to this. When my site gets knocked out of the #1 slot on MSN, it's ALWAYS been because one of their bots got banned. I 'unban' them, and a couple of days later, I'm #1 again.

So, you're thinking along the right track IMHO, except it might be YOU that's doing the banning; not the other way around. Be nice to search bots, even if they're acting naughty!

Posted: Mon Jan 17, 2005 9:54 am

thanks for the tips! questions though....

what are "MSA Stats" sorry!! Do i access these through my control panel?

when you say you check your IP ban list, do you mean you check it through the admin control panel under "IP Ban"?

Where can i get the "who is where" block? Thanks.

Posted: Mon Jan 17, 2005 5:55 pm

Maybe your site has not enough other partnerlinks or not enough content. But content isn`t so necassary.
Hwo many results did`t you get from Google and have the results any describtion:
Check
site:www.yoursite.com

and perhaps check the html code:

validator.w3.org/check/referer

and check your logfiles maybe with awstats (but not the modul for nuke !) for msnbot and yahoo:

65.54.188.144 - - [17/Jan/2005:00:00:53 +0100] "GET /content.html HTTP/1.0" 200 21738 "-" "msnbot/0.3 (+http://search.msn.com/msnbot.htm)"

68.142.250.59 - - [17/Jan/2005:23:57:21 +0100] "GET /ftopict-1044.html HTTP/1.0" 200 7828 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

Posted: Tue Jan 18, 2005 2:48 am

candy wrote:

...what are "MSA Stats" sorry!!

...when you say you check your IP ban list, do you mean you check it through the admin control panel under "IP Ban"?

Where can i get the "who is where" block?

By 'MSA Stats' I meant MS-Analysis Statistics: http://www.matyscripts.com/ Wink

Yes, I check my IP Ban list in the admin panel, or I simply look at the scrolling 'Ban Block' on my site...

And, you can get 'Who-is-Where' here: Only registered users can see links on this board! Get registered or login!

Posted: Tue Jan 18, 2005 3:01 am

Here's an example of 'Who-is-Where' in action:

(mod'ed, of course - truncated module names and IP's)

Can you tell which 'visitors' are bots? ROTFL

Here's what MSA Stats look like:

Posted: Tue Jan 18, 2005 4:20 pm

thanks so much for the info! i'll have to employ these methods soon!