How Was This Blocked?

Posted: Tue Dec 07, 2010 9:10 am

Guardian2003 wrote:

Just to clarify; I presume that because your Harvester menu is empty, the table is also? ( _nsnst_harvesters )

You presume correctly and hence the mystery and YES, the DB is the correct one - been there done that once.

Referrers table is populated but I don't think that has any bearing on anything related here.

Subject Matter Expert Joined: Feb 23, 2004 Posts: 358

Like I posted a few days back, I have a huge amount of UA's with that exact string in it, and they are not being blocked.

You said your files are default, and your database tables shouldn't be an issue...

At this point there's only one way to deal with this, and it's from THAT server, since all the info you have provided us makes it impossible for us to duplicate the problem.

You will need to go into the code. There is a point where the UA is assigned a variable, and a point where that variable is analyzed and either trapped or allowed to go on its merry way. You must write some debugging code to intercept that UA. You can do a

Code:

if (is_admin($admin)) {


    $the_variable = 'the UA you want to force it to be';


}

Now whatever's being triggered should get triggered every time an Admin requests a page. Add more code that is specifically looking at the other variables involved, especially arrays in that section. Using an "admin" branch of code like that can let you do a lot of things that never disturb anyone else (assuming the site is live).

Eventually you are going to start seeing exactly what the code is seeing, and you can start feeding us better information, unless of course the problem just becomes obvious to you and you fix/patch it...

- Keith

Posted: Tue Dec 07, 2010 2:40 pm

Thanks for the suggestions, etc., will keep it on the front burner. I removed the _harvesters table and replaced it with another, unmodified one from one of my test domains. So far, no problems encountered ... yet - there's always a gremlin hiding around the corner but if the problem doesn't surface in the next few hours then it may just be fixed.

Cheers

Posted: Tue Dec 07, 2010 5:10 pm

Going through this part of the code in Sentinel, and running across something either Raven or someone else needs to clarify...

The table _nsnst_harvesters is called and loaded in Sentinel Admin Harvester section, and you can add/delete items from it. If you EMPTY the table from phpMyAdmin, and re-access the section in Sentinel Admin, it will report that there are no Harvesters.

All looks good... except...

When you output the Harvesters list in the sentinel script, there they are again! In tracking down where this list is coming from, I found that in _nsnst_config, there exist a config name of list_harvester, and a config value of - all of the items I just emptied from the _nsnst_harvesters table! Upon further review and searching through code, I cannot find any reference to _nsnst_harvesters other than the admin script that adds/modifies/saves etc.

Why are there two separate lists, and why doesn't sentinel load the one that is actually configurable by the Admin?

So dad... even though your _nsnst_harvesters table is empty, or whatever you have done to it. Check your _nsnst_config table and see if there is a second list under list_harvesters.

This one has me scratching my head...

unless somewhere in a script I haven't looked in yet is populating one list with another.. but that doesn't make sense.

The harvester check section of the code specifically creates an array from ab_config['list_harvesters'], which means whatever's in the _nsnst_harvesters table has no effect whatsoever on anything, as far as I can tell.

halp!

- Keith

Posted: Tue Dec 07, 2010 5:47 pm

Thanks, this "may" be a problem .. However, in the Blocker Configuration I have for "Harvesters" => Email Admin .. nothing blocked or sending a default page, etc. However, I still get an email, such as a few minutes ago:

Code:

Created By: NukeSentinel(tm) 2.6.03


Date &amp; Time: 2010-12-07 17:38:51 CST GMT -0600


Blocked IP: 119.63.198.80


User ID: Anonymous (1)


Reason: Abuse-harvest


String Match: Baiduspider

Why does it say "Blocked IP" if I have it set for "Email Admin" and not "Blocked", etc. ? also note the spelling of "Abuse-harvest", I purposely spelled "harvest" in lower-case "h" in the blocker configuration to see if the email is actually coming from where it is supposed to be.

Is the IP actually being blocked? I don't think so after checking the blocked IP tables.

Very confusing to say the least. I'll check the Sentinel Admin stuff you mentioned.

Cheers

Posted: Tue Dec 07, 2010 5:55 pm

Quote:

Check your _nsnst_config table and see if there is a second list under list_harvesters.

Indeed there was and the second list is the one that is manually added when you add an agent or partial agent string.

And lo and behold, in that list was "CC" which was/is causing all the trouble!!!!

Cheers

Thanks for the heads up, I never would have thought to check that list in the _nsnst_config under list_harvesters.

Cheers

Posted: Tue Dec 07, 2010 6:57 pm

Nice!

Well, now we need to figure out why there's duplicate lists.. because there's also lists in _nsnst_config for referers, strings, etc... so somewhere a change in how this all works was implemented, but some of the details didn't get changed.

An admin should be able to access these lists, make changes, and the script execute those changes. As for now, nothing you do to the Harvester list from the Admin menu makes any changes. You can only toggle it on or off. Will wait for Raven or someone else to shed light there. I could fix it now, but I'll wait to find out which list they intended the script to use.

Glad you at least found and fixed one part of the puzzle! Happy to help...

- Keith

Posted: Tue Dec 07, 2010 7:20 pm

Excellent find! I had not spotted that..

Posted: Tue Dec 07, 2010 7:46 pm

Quote:

Excellent find! I had not spotted that

And that now answers the question where the manual UA changes go like when you click on an agent to flag it and only pick out part of the string and save it. THAT is where it goed, in that second list. Hah!!! Wink

Remember my query a while back regarding the case sensitivity of UA strings when you save 'em? Now we know.

Cheers

Posted: Tue Dec 07, 2010 9:06 pm

Now I agree that there should not be two locations for the harvesters, etc, but I also can not find an instance where they are both not updated.

Posted: Tue Dec 07, 2010 9:10 pm

Palbin wrote:

Now I agree that there should not be two locations for the harvesters, etc, but I also can not find an instance where they are both not updated.

The two lists combined into the one list in the table with the manual additions at the top and yes - updated each time one is added/removed, etc.

Posted: Tue Dec 07, 2010 10:38 pm

Where are we now? Are we saying that there's supposed to be two lists, and they are supposed to be sync'd? I haven't gone through the Admin scripts yet to see if that's the case.. all I know is that I emptied my _nsnst_harvesters table, and the ones in the config table remained. If they are supposed to be sync'd, there should be code in there to keep them sync'd. Obviously somehow dad got a rogue CC in the config value, but it wasn't in the other table, so this sync process, if it exists, needs some tweaking.

- Keith

Posted: Tue Dec 07, 2010 10:47 pm

What I did was to empty/drop the _harversters table and repopulated it with a clean, default copy from a test domain. After inspecting the list_harvesters in the config table, the new list was there as well as the manual entries from before the empty and repopulate.

To conclude .. Emptying the harvesters table does NOT also remove the manual entries in the config list-harvesters list. But since I hadn't inspected the list-harvesters in the config table prior to Keith's post(s), I can't be sure that emptying the harvesters table also removed those entries from the list. But I would assume that the two are separate and not sync'd - emptying one has no effect on the other.

Posted: Tue Dec 07, 2010 10:51 pm

ok, so a review of one of the admin scripts that saves harvester data reveals that, indeed, if the admin viewable portion is updated, it is saved, then merged into the config value, re-sorted, and then re-written to the config table.

So there is two lists, and an attempt to keep them sync'd. That sync method isn't sound, as it only sync's one way and therefore depends on the "perfect world" theory. 99% of the time, it's gonna work...

I'm curious as to the logic of maintaining two lists? Is this less MySQL overhead to pull these values from the config table and travel around the script with those values in an array?

- Keith