Google being Blocked ????

Client Joined: Dec 04, 2004 Posts: 757

Hello all, I am finding that google seems to sometimes being blocked I found this out by the site diagnostic tool in google adsense etc
the basic robots.txt is just the latest default one and I have added nothing to it.
any idea`s all
cheers
mrix

registered · Posted: Fri Jul 20, 2007 5:47 pm

You might want to see if they are in the Flood Blocker. You might also want to log into your google account and set the crawl delay to be something quite a bit larger than your flood blocker time interval.

That is the only thing that I can think of when you say "sometimes being blocked".

Posted: Sat Jul 21, 2007 3:13 am

I did activate flood Blocker and at the moment its set on 1..... so are you saying maybe at times googlebot gets blocked at times through flood delay etc ?
Bascially I hope so that would answer my problem.
many thanks for the help
also how do I actually turn flood blocker off as it just has a drop down of 1 to 5 etc ?

mrix

Posted: Sat Jul 21, 2007 7:53 am

Yes, that is what I am saying. 1 second is very aggressive, so, you could be blocking much more than you want. But, the key is to add a Crawl-Delay parameter in your robots.txt (Google for it if you are not familiar with that) AND, most importantly for Google, you have to sign in to your Google account and set the google spider crawl delay to something above your flood blocker setting - i'd recommend double it).

Google does NOT obey the Crawl-Delay directive.

BTW, there are several threads here about this topic if you want to read more about it.

Posted: Mon Jul 23, 2007 3:50 pm

Hello again I have adjusted varius settings but still getting google being blocked this it what it shows in whats being blocked

admin. php? op= hreferer Robots.txt File 21 Jul 2007 1
admin. php? op= ABTrackedMenu Robots.txt File 21 Jul 2007 1
admin. php? op= ABTemplate Robots.txt File 21 Jul 2007 1
admin. php? op= ABImportIP2Country Robots.txt File 21 Jul 2007 1
admin. php? op= ABImportBlockedRange Robots.txt File 21 Jul 2007 1
admin. php? op= ABImport Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 800 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 6800 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 6600 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 6400 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 6200 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 6000 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 600 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 5800 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 5600 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 5400 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 5200 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 5000 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 4800 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 4600 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 4400 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 4200 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 4000 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 400 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 3800 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 3600 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 3400 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 3200 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 3000 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 2800 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 2600 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 2400 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 2200 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 2000 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 200 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 1800 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 1600 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 1400 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 1200 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked&min= 1000 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateTracked Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateExcludedRanges&min= 200 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateExcludedRanges Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateBlockedRanges Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateBlocked&min= 200 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryUpdateBlocked Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryOverlapCheck&min= 50 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryOverlapCheck&min= 250 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryOverlapCheck&min= 200 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryOverlapCheck&min= 150 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryOverlapCheck&min= 100 Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryOverlapCheck Robots.txt File 21 Jul 2007 1
admin. php? op= ABIP2CountryList Robots.txt File 21 Jul 2007 1
admin. php? op= ABExcludedMenu Robots.txt File 21 Jul 2007 1
admin. php? op= ABExcludedList Robots.txt File 21 Jul 2007 1
admin. php? op= ABDBStructure Robots.txt File 21 Jul 2007 1
admin. php? op= ABDBRepair Robots.txt File 21 Jul 2007 1
admin. php? op= ABDBOptimize Robots.txt File 21 Jul 2007 1
admin. php? op= ABDBMaintence Robots.txt File 21 Jul 2007 1
admin. php? op= ABCountryList Robots.txt File 21 Jul 2007 1
admin. php? op= ABConfigRequest Robots.txt File 21 Jul 2007 1
admin. php? op= ABConfigReferer Robots.txt File 21 Jul 2007 1
admin. php? op= ABConfigFlood Robots.txt File 21 Jul 2007 1
admin. php? op= ABConfigFilter Robots.txt File 21 Jul 2007 1
admin. php? op= ABConfigAdmin Robots.txt File 21 Jul 2007 1
admin. php? op= ABConfig Robots.txt File 21 Jul 2007 1
admin. php? op= ABBlockedRangeMenu Robots.txt File 21 Jul 2007 1
admin. php? op= ABBlockedRangeClear Robots.txt File 21 Jul 2007 1
admin. php? op= ABBlockedIPMenu Robots.txt File 21 Jul 2007 1
admin. php? op= ABBlockedIPClear Robots.txt File 21 Jul 2007 1
admin. php? op= ABBlockedIP Robots.txt File 21 Jul 2007 1
admin. php? op= ABAuthListScan Robots.txt File 21 Jul 2007

registered · Posted: Mon Jul 23, 2007 5:22 pm

Do you have a link to your admin.php on your site somewhere? In general you don't want to expose that, I would think.

Posted: Tue Jul 24, 2007 6:36 am

I would agree with Gremmie, you must have an open link to the admin file somewhere.
One thing google does do well is stick to the directives in robots.txt and that prohibits it from crawling admin.php

Looking at your post, it looks as though this is what google has done. It has found links it shouldnt have but it clearly shows that it has stopped itself due to robots.txt directives.

Posted: Tue Jul 24, 2007 10:05 am

No I havnt changed anything on my site like that in years.
This is so anonying and damaging my site sob sob
Cheers anyway
mrix

Posted: Tue Jul 24, 2007 10:28 am

Well you do want robots blocked when they try to crawl your admin links, so your robots.txt file is doing its job. How those robots got your admin links should be looked into though.

Posted: Wed Jul 25, 2007 5:44 am

mrix, how is this "damaging" your site? These directives are in your robots.txt file (or one strictly for admin.php) and Google appears to be honoring your request for them not to crawl those links... Confused

How those links were ever cached in the first place is beyond me. Search Engine crawlers can't just make things up. They have had to have had at one point in time been a link that they followed...

Posted: Wed Jul 25, 2007 5:46 am

Hi is this just telling me the admin pages have been blocked ?
Cheers
mrix

Posted: Wed Jul 25, 2007 6:26 am

It is telling you that Google is following your "orders" to not crawl anything that has admin.php in it. This is perfectly acceptable. How they got the original links to begin with is the real question mark here, but not sure it is worth digging into if you are certain that you do not currently have any admin links available to anonymous any longer.