Google Booted me - its a sad day

Client Joined: Dec 04, 2004 Posts: 757

Hi all, I have been slowly updating sentinal over a period of about a year, I updated this time and was receiving emails that google bot was getting blocked so I added some google ip`s to the protected bit in sentinal. Unfortunately today I find my main keyword that has been ranked 2nd to top in google is know where to be seen anymore ouch!.
why has sentinal decided to do this now I have been ok for like over a year it also tries to block yahoo and msn. has anyone got the ip ranges for just these search engines please. Sad

, also I am a it worried now about my robots.txt could someone give me an idea what I should have in my robots.txt to allow the 2 search engines in?
thanks for any help
mrix Sad

Sells PC To Pay For Divorce Joined: Posts: 5661

to get the ip ranges you can view this...
http://www.iplists.com

And as for your robot.txt ,
just start your file with..

Code:




User-agent: Mediapartners-Google*


Disallow:

Posted: Sun Sep 11, 2005 4:34 am

Hi I was looking at the yahoo google and msn ip`s but see masses of them to be honest I am not sure what I need and what I dont need?
Cheers
mrix

Posted: Sun Sep 11, 2005 4:41 am

well in my case i only have google in protected range ,others are not banned....
thing is that you have to look up the ip range ...

Posted: Sun Sep 11, 2005 5:36 am

To be totally honest I am at a loss what you are saying here, all I want is to be indexed by most of the search engines at the very least Google MSN and Yahoo I am suprised there is not basic info anywhere to allow these engines bit not only that surely sentinal should add these as default?
could you please tell me if my robots.txt file looks ok this is the contents at the moment

User-agent: Mediapartners-Google*
Disallow:

Disallow: /admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

would you say this is ok?
I do appriciate your time
thanks
mrix

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

NukeSentinel does protect these by default. Something else appears to be going on because I, as well as many others, use NS and am properly indexed by the search engines.

Posted: Sun Sep 11, 2005 8:25 am

I see, all I know is that I have been ok all this time and now since the update of sentinal I have had these problems I have not added anything to my site apart from sentinal for a few months. its really dissapointing to be honest to have my main keyword kicked from google as you can imagine Sad

.
Cheers
mrix

registered · Regular Joined: Feb 16, 2005 Posts: 67 Location: Canada

try changing the robots txt to:

User-agent: *
Disallow: /admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

Not a big diff but better syntax ps. This wont help you with Sentinel, sorry

registered · Posted: Sun Sep 11, 2005 11:02 pm

When you get an email from NukeSentinel(tm) telling you it blocked a Google search what does it have in it. We can sit here a guess all day at why it now blocks them and didn't before. My best guess is your Harvester list has something in it that is triggered by the search engine user agent string.

Posted: Sun Sep 11, 2005 11:06 pm

On the ip list ( http://www.iplists.com/google.txt ), I ran the first ip thru an IP Whois and it has nothing at all to do with google:

Code:

inetnum:      164.69.0.0 - 164.71.255.255


netname:      FUJITSU


country:      JP


descr:        Fujitsu Ltd.


admin-c:      DY215-AP


tech-c:       DY215-AP


status:       ALLOCATED PORTABLE


mnt-by:       MAINT-JPNIC


mnt-lower:    MAINT-JPNIC


changed:      **********@apnic.net 20050408


source:       APNIC

Posted: Mon Sep 12, 2005 12:04 am

Well as I said a couple of days ago I was getting emails explaing google had been blocked these messages went to my junk email and now are deleted unfortunately and I havnt any since, my site under the keyword "Sea Fishing" has ranked no2 in google for 2 years and now I find my site even though it has a 5 rank is know where to be seen? it appears its only my index page thats not getting cached as most of my others are not sure if this is a clue? but it all happened a couple of days adter I installed Sentinal. but like i say I am also getting blocks from yahoo Slurp and MSN but they only seem to be my links and download pages for some reason? I have some details here where a page is getting blocked from one of my 2 sites.
Cheers
mrix

Date & Time: 2005-09-11 23:45:35 EDT GMT -0400
Blocked IP: 207.46.98.138
User ID: Anonymous (1)
Reason: Abuse-Script
--------------------
User Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)
Query String:
www.battlefield-2.biz/modules.php?name=Downloads&d_op=viewdownload&cid=2(\"derby=titleA
Get String:
www.battlefield-2.biz/modules.php?name=Downloads&d_op=viewdownload&cid=2(\"derby=titleA
Post String: www.battlefield-2.biz/modules.php
Forwarded For: none
Client IP: none
Remote Address: 207.46.98.138
Remote Port: 44585
Request Method: GET

Posted: Mon Sep 12, 2005 1:16 am

The reason is easy for me to see. It is tring to index pages with ( and ) in them. This is a leading character found in scripting attacks.

Check all your Download/Web Links titles for those two characters and replace them with something like [ and ].

Posted: Tue Sep 13, 2005 1:17 am

I only actually see them characters in a description of a link which I did change and still am being blocked I clicked on the link to the downloads page in the email to only see those brackets used by default in the actual downloads modules, where it ses search by (A/B) (C/D) etc ,surely this must be sentinal playing up? I am suprised no one else is getting any problems with this sentinal upgrade?
Cheers
mrix

Posted: Tue Sep 20, 2005 2:37 am

This is getting me down I find that I can submit a Gogle Sitmap to my subdomain website no problem at all but find I cannot submit a site map for my main site it comes up as an error??? this is stuffing my site I think in some way sentinal is blocking it as I havnt done anything to my site in months just the sentinal upgrade.
Cheers
mrix

registered · Posted: Tue Sep 20, 2005 5:43 am

Before you do anything else.

Do a "site:www.yoursite.com" in the google search box. Are all your pages still indexed?

Do you still rank for secondary keywords?

It's quite possibly a coincidence that you lost your search engine listing and installed sentinel within the same time frame.

Do you have msanalysis installed? If you can give the site details then I can take a look for you.

Tony

Posted: Tue Sep 20, 2005 6:23 am

Hello I have had my site high ranked under the term "Sea Fishing" well second place in google for about 2 years etc. I installed well upgraded to the new sentinal and started to recieve blocking emails that I never had before from like Yahoo Slurp MSN and google. then about 3 days later I noticed a massive drop in my site hits from like 700 to under 300 unique hits a day etc so I checked my pages and found my index was know longer cached? I tried other pages but they were still cached? but after that day by day almost all my pages have been lost...ouch, I`ll install this msanalysis and get back to you .... many thanks for your time on this
Cheers
mrix

Posted: Tue Sep 20, 2005 12:35 pm

Taking a quick look at your source code I see you have double <he ad></he ad> tagsin the index, and in the forums section the source shows double head tags then it has the closing </b ody> and </ht ml> tags with code after, like here:

Quote:

</T R>
</T ABLE>
</B ODY>
</H TML>
<t able width="100%" cellpadding="0" cellspacing="0" border="0" align="center">
<t r valign="top">

Then further down the page another <he ad> tag:

Quote:

You can almost bet the search engine is only reading the first section of the page, if at all.

Checking with the W3C validator shows an error for the forums page:
This page is not Valid (no Doctype found)!

My guess for the overall problems with search engines is not Sentinal but inccorect page layout coding. In the past maybe Google has been more tolerant of errors but it changes all the time. Check for yourself: http://validator.w3.org/

By the way, I really like your site!