Why was this action blocked?

Involved Joined: Dec 28, 2003 Posts: 276 Location: Israel

Any idea why this happened? all the user did (according to what I can see), was running index.php under the main site. Or am I missing something?

Date & Time: 2004-11-08 06:41:57
Blocked IP: 80.230.116.*
User ID: not registered (1)
Reason: Abuse-Harvest
String Match: microsoft url control
--------------------
User Agent: Microsoft URL Control - 6.00.8862
Query String: hagigim.com/index.php
Forwarded For: none
Client IP: none
Remote Address: 80.230.116.151
Remote Port: 21059
Request Method: GET

Posted: Mon Nov 08, 2004 6:32 am

This happened 10 minutes ago. I've unblocked the ip, thinking it was safe, and now I get this:

Date & Time: 2004-11-08 07:23:35
Blocked IP: 80.230.116.*
User ID: not registered (1)
Reason: Abuse-Harvest
String Match: microsoft url control
--------------------
User Agent: Microsoft URL Control - 6.00.8862
Query String: www.hagigim.com/modules.php?name=Forums&file=viewtopic&p=22235
Forwarded For: 80.230.116.151
Client IP: none
Remote Address: 80.230.116.151
Remote Port: 47880
Request Method: GET

Am I being hacked or what?!

PS: am using Sentinel 2.1.0, for about a month in which it blocked a Chinese IP, and nothing since till today. Could anyone explain?

registered · Posted: Mon Nov 08, 2004 6:45 am

ring_c wrote:

Am I being hacked or what?!

It's hard to tell without looking through your logs, but most likely it's a spam bot looking for a formmail to send out spam. Either that or it's a snoop bot trying to collect email addies from your site. It just depends on how it's setup...

Posted: Mon Nov 08, 2004 8:15 am

microsoft url control is not listed in the Sentinel Harvesters list for nothing. In my opinion I would add the IP back in to your blocked list.

Posted: Mon Nov 08, 2004 8:57 am

Anytime you see Reason: Abuse-Harvest visit Ravens Only registered users can see links on this board! Get registered or login! page and paste the User-Agent in and click submit to see if it matches a default trapped user-agent.

In this case you would have seen:
Agent: microsoft url control is trapped by this Harvester entry: microsoft url control

Posted: Mon Nov 08, 2004 9:02 am

sixonetonoffun wrote:

Anytime you see Reason: Abuse-Harvest visit Ravens Only registered users can see links on this board! Get registered or login! page and paste the User-Agent in and click submit to see if it matches a default trapped user-agent.

In this case you would have seen:
Agent: microsoft url control is trapped by this Harvester entry: microsoft url control

Now, this was Chinese to me. sorry.
Anyway, should I realy re-enter the IP to the list of blocked addresses?

Posted: Mon Nov 08, 2004 9:13 am

He was saying that Raven has an "agent inspector" here on this site (See the menu block!) where you can copy in the "Agent Reason" information that was emailed to you.

It will then tell you if this was a valid agent blocking function.

Quite frankly though, this looks like a harvest agent slurping your website to steal email addresses and images, so yes, this was a valid action by NukeSentinal that protected your site from abusive "Harvest" functions. NukeSentinel did not block a user, but rather an automated process being used by someone to steal information.

Just as a heads up, your site is not compatible with Firefox or Mozilla browsers which I would consider to be a serious issue.

Hope this helps.

Posted: Mon Nov 08, 2004 9:20 am

64bitguy wrote:

Just as a heads up, your site is not compatible with Firefox or Mozilla browsers which I would consider to be a serious issue.

Hope this helps.

Thanks. I'm using Php-Nuke v6.7, how can I fix it then?

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

Running that agent string in the Agent Inspector you get

Agent: microsoft url control is trapped by this Harvester entry: microsoft url control

If you want to allow it then delete that entry from your Harvester settings in NukeSentinel.

Posted: Mon Nov 08, 2004 9:58 am

Raven wrote:

Agent: microsoft url control is trapped by this Harvester entry: microsoft url control

But what does it mean? Is it harmfull/harmless/good/bad?
I'm cluless...

Posted: Mon Nov 08, 2004 10:10 am

It is a known harvester that is not trusted/wanted. Do a google search for more information.

Posted: Mon Nov 08, 2004 11:04 pm

Just add the IP back to the ban list . You don't need that User-Agent on your site... Wink

Posted: Tue Nov 09, 2004 12:02 am

Raven wrote:

It is a known harvester that is not trusted/wanted. Do a google search for more information.

Thanks, Raven. I did.
Is there a way to deny this agent to run? Or maybe Sentinel is the best solution, and I should let it to the job like it did this time?

Posted: Tue Nov 09, 2004 12:09 am

Just let Sentinel stop it. You could use .htaccess to deny it also. That way it never makes it to your site.

Posted: Tue Nov 09, 2004 12:31 am

Raven wrote:

Just let Sentinel stop it. You could use .htaccess to deny it also. That way it never makes it to your site.

Do you think I should? If so, could you please guide me how to do it?

Posted: Tue Nov 09, 2004 2:14 am

I've just got this one. I guess I should check your agent inspector... Anything you might add?

Date & Time: 2004-11-09 02:08:05
Blocked IP: 209.167.50.22
User ID: not registered (1)
Reason: Abuse-Harvest
String Match: linkwalker
--------------------
User Agent: LinkWalker
Query String: www.hagigim.com/index.php
Forwarded For: none
Client IP: none
Remote Address: 209.167.50.22
Remote Port: 45972
Request Method: GET

Posted: Tue Nov 09, 2004 2:24 am

ring_c wrote:

I've just got this one. I guess I should check your agent inspector...

I got this:

Code:

Agent: LinkWalker is trapped by this Harvester entry: linkwalker

Now, what does it tell me? How can I tell if it's good or bad?

Posted: Tue Nov 09, 2004 2:27 am

Ok, being googling a little, and found a good (hopefully) database of web robots, which might be helpfull to some of us, here:
http://www.robotstxt.org/wc/active/html/type.html

There, I could find a link to LinkWalker's tech page, here:
http://www.seventwentyfour.com/tech.html

Yet again, I can't realy tell if this kind of harvest is good or bad. how do you do that?

Client Joined: Jul 18, 2003 Posts: 977

Harvest is generally bad. There is a reason they are in the Sentinel harvest blocker list. If the makers of Sentinel deem they are bad then I will listen. Anything in that list... leave it. Block 'em all. Crawlers are a different story, sites want some crawlers to increase traffic. The good crawlers are not in the list...

Posted: Tue Nov 09, 2004 12:09 pm

There are harvester lists if you google for them. We include the 'master' list, if you will. That's why YOU make the final decision Wink

Posted: Tue Nov 09, 2004 1:41 pm

I spent an entire day reviewing the Harvester List provided by NukeSentinel when I first loaded it. I did this because I've been maintaining my own harvester list in my existing .htaccess file as well as some of the other Programs that I used in conjunction with my site.

I was rather surprised that NukeSentinel's list not only contained all of the standard Nuke recommended to block harvesters, but also a few others that I hadn't heard of yet. On the other side of the coin, I had a few on my list that were not yet in NukeSentinel, which I promptly added.

My feeling (after exhaustively researching the list of included harvesters for blocking) is that the list is pretty extensive and thorough. It blocks the majority of Harvesters that are designed to extract email addresses, images and other proprietary information from websites. For those not familiar, this information would most likely extracted with abusive intentions to steal or hotlink your resources and to SPAM you and your users. The Federal Trade Commission estimates that well over 80% of all SPAM is directly attributable to website harvesting. This fact alone should be enough to convince people to use this valuable NukeSentinel feature to block all harvesters.

See: http://www.ftc.gov/bcp/conline/pubs/alerts/spamalrt.htm

The FTC replicated this test in 2004 with the same results and still attribute 85+% of all SPAM to originate from automated harvesters employed by Spammers.

See: http://www.oecd.org/dataoecd/26/54/26516855.pdf

While Raven points out that "YOU make the final decision" on how to deal with harvesters (as well as the list you keep inside NukeSentinel) I would recommend from personal experience that you maintain the existing list of abusive harvesters and even add to that list as you discover additional abuses.

As pointed out above, harvesting is different than "spidering" or "crawling" your site. My experience has been that harvesting always results in abuse, whereas "spidering" or "crawling" is simply a method used by search engines to index data for productive purposes. Of course this also is not always the case, as there are also abusive robots and spiders. Again, it is up to YOU to decide who you want spidering or crawling your site.

With a few tools, you'll often find that SPAM is a direct result of an unknown harvester "having its way" with your site and through your data.

One of the tools I employ is Visual Route ( See http://www.visualware.com/personal/products/visualroute/index.html ) to determine where SPAM is really coming from. With a little dilligence you can figure out what harvester they are using and block it. You can also employ some creative mail management techniques to prevent further abuse. I (For example) "blacklist" the abuser in my server hosted Spam Assassin and "bounce" the emails back. I also ban their IPs and report the abuse to the Host and ISP as well as Operation Web Snare. (See http://www.fbi.gov/cyberinvest/websnare.htm ) If you attempt abuse on my site, chances are pretty good that the FBI will be monitoring your activities in short order. A few of the recent SPAM busts are a good example of successes. I believe we will be seeing many more arrests in the near future as CAN-SPAM ( See http://www.ftc.gov/bcp/conline/pubs/buspubs/canspam.htm ) starts to be enforced.

The long and short of it? Use NukeSentinel to protect your resources from Harvesters that would abuse you and your users and Add to the list, don't reduce it. Finally, take actions to protect email addresses on your site.

1) Never publish an email address ANYWHERE on your site.
2) Never include non-spam proof email addresses in any program you create. (use "joe at joes dot com" and not joe@joes.com")
3) Enable phpbb's function to "User email via board" (Admin/Forums/Configuration) which will hide user email addresses normally visible in "Userinfo" and force mail sent to your users to go via your site and "Private Messenging"
4) Maintain a good Harvester List in NukeSentinel and .htaccess to keep them out of your data.

Hope this helps.

Posted: Tue Nov 09, 2004 1:52 pm

Excellent! And I would Enable phpbb's function to "User email via board" and have tried umpteen times. But, alas, to no avail. It does not work and I have had others look at it.

Posted: Tue Nov 09, 2004 2:09 pm

hmmmm... It used to work here... In fact this is the site where I discovered I didn't have mine enabled. I dunno what happened in that regard. I'm using 2.0.10, but yours for some reason is reporting the old 2.0.6, though I thought it had been patched to a later version.

One thing I have noticed is that your GT link to profiles is different than mine. My link to profile data is a little shorter (SITENAME-forum-userprofile-USERID#.html) and I have disabled the next layer functions in GT (Removed the in/out definitions) and it's working fine. Basically, I didn't want that data GT'd anyway, that's why I did that.

I'm more than happy to share my .htaccess data and in/outs if you think it might help.

Posted: Tue Nov 09, 2004 4:02 pm

I just reactivated the email via board. Try to send me an email.

Posted: Tue Nov 09, 2004 4:29 pm

Sent, but I should note that a copy of the sent message does not appear in either my sentbox or outbox, though that option was checked