MSN Bots ?

Client Joined: Dec 04, 2004 Posts: 757

Hello all, just recently I have had loads of msn bots on my site Question

Basically I can have about 20 members on and around 400 to 600+ visitors Crying or Very sad

when checking the ip`s most of the visitors are MSN bots.
Has anyone had this issue and if so what can I do to help it Question

the last thing i would want to do is block the MSN Bots all together as you can imagine.

thanks fo any idea`s all
mrix

Posted: Tue Mar 18, 2008 10:20 am

The Inktomi/Slurp bot is reknowned for this behaviour - just don't try banning them as they will come back with the second wave but I have not see this with MSN bots.

To save your resources and bandwidth, in your robots.txt file you can add the following directive for Yahoo/Slurp bot

Code:




User-agent: Slurp


Crawl-delay: 10

where 10 is the number of seconds between requests.
For the MSN bot you can use;

Code:

User-agent: msnbot


Crawl-delay: 10

However, to cover for all bots that adhere to the crawl delay directive you might want to use a blanket directive like;

Code:

User-agent: *


Crawl-delay: 10

Posted: Tue Mar 18, 2008 10:29 am

Thats great I`ll give it a go
cheers
mrix