robots using bandwidth

geoff_bell · Hangin' Around Joined: Dec 07, 2006 Posts: 41

i have two accounts with a host. 100mb space and 1gb bandwidth. they are adequate to run three nuke sites as i use photobucket.com to host my images.

one account exceeded the bandwidth last month after only ten days and the second account is presently sitting close to 90% after about half a month.

i have altered the robots meta tag line in my meta.php file to read "INDEX, NOFOLLOW" which i hope will help prevent this in the future, even though i would have preferred not to as "google is your friend" is a popular internet quote.

in awstats the robot is unnamed (only referred to as "crawl"), so i cannot ban it using robots.txt as, unless i ban all robots mentioned with "crawl" in their names.

any ideas?

Susann · Posted: Thu Dec 14, 2006 6:02 pm

1 GB wouldn´t be enough for my nuke site.However, you need to check your logfiles to find out more about Unknown robot (identified by 'crawl').In awstats is beneed the hits also the date and the time. So it should be an easy task to check for that and in NukeSentinel are several ways to ban and you can ban directly in the .htaccess.
Bad bots ignore the robots.txt.
Here is a list:

Only registered users can see links on this board!
Get registered or login to the forums!

Metatags:

Only registered users can see links on this board!
Get registered or login to the forums!

Guardian2003 · Posted: Thu Dec 14, 2006 7:44 pm

Yes, using Sentinel to check for the IP's most frequently visiting your site might definitely help you track that one down.

geoff_bell · Hangin' Around Joined: Dec 07, 2006 Posts: 41

i don't have sentinel on this account Embarassed

i have checked awstats and tracked it down - it accounted for 750mb of traffic!

and banned it. considering banning most bots except google