Search Engines get blocked?

Client Joined: Dec 04, 2004 Posts: 757

Hello all, I cant understand why search engines like msn get blocked from my site? I am running the latest sentinal. How do allow msn to search my site? I would have thought sentinal would allow it by default? I am wondering now how many more search engines maybe being blocked etc?
thanks for any help all
mrix

Posted: Thu Feb 22, 2007 4:46 pm

What part of the site are they trying to access?

What reason is sentinel giving for blocking the SE?

Posted: Thu Feb 22, 2007 5:27 pm

This one block

User Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)
Query String:
/modules.php?name=Downloads&d_op=viewdownload&cid=1(\"derby=titleA
Get String:
/modules.php?name=Downloads&d_op=viewdownload&cid=1(\"derby=titleA
Post String: /modules.php
Forwarded For: none
Client IP: none

registered · Posted: Thu Feb 22, 2007 6:27 pm

mrix, no, what was the block reason?

I have had a few like this recently from msnbot. Its almost like they have changed something and are possibly ignoring the crawl-delay statement that I have in my robots.txt. This used to work fine. However, it is not ALL of the msnbots. So, maybe they just have bug causing one or two to go "astray".

Mine were tripped with the flood blocker.

Posted: Thu Feb 22, 2007 6:29 pm

Oh, crap, I forgot to mention that you can keep them from getting blocked by adding them to your Protected Ranges. Hey, new IP ranges are assigned all the time. So, these are most likely new ones that you have not seen before. You have to keep adding these as you find them.

Posted: Thu Feb 22, 2007 6:30 pm

Hello yea sorry this is the top section of that
Blocked IP: 65.55.212.235
User ID: Visitor (1)
Reason: Abuse-Script

I also at time get some yahoo bots blocked also

cheers
mrix

Posted: Thu Feb 22, 2007 6:32 pm

I see so whenever a search engine gets blocked I can add it to the Protected Ranges
ok that sounds good to me
thanks for the help all
mrix

Posted: Thu Feb 22, 2007 6:54 pm

Ok, that is different. Do you by chance have parenthesis in your download title? If so, i'd get rid of them.

Posted: Thu Feb 22, 2007 6:56 pm

Quote:

have parenthesis in your download title?

sorry I have not idea what you mean could you explain a bit more
thanks for the help
mrix

Posted: Thu Feb 22, 2007 7:00 pm

Well, that query string for one is looking really odd to me. Have you done anything recently to your downloads, such as install a different downloads module?

But, do you see this here: (\"derby=titleA

The backslash "\" is simply there in the email (added prior to being sent out), but I am concerned by this:

(derby

Does your download title start off with a left-parenthesis? Download title is the name that you gave it in the downloads admin.