Can't stop harvesting with Sentinel 2.0 and fetch-it

Posted: Sat Aug 07, 2004 8:34 am

Hi all,

Great work with Sentinel 2.0, but I seem to have a problem. Confused

Harvesting of downloads still going on, the files are directly requested without any problem, no fetching neccesary or blocking by Sentinel. Embarassed

How can I change this to be protected by Sentinel and that Fecht-it needs always to be used. Crying or Very sad

Thanks to all,

Fred

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

Directly downloading is not the same as harvesting. Sentinel will only stop harvesting IF the Agent that is doing the harvesting is in the Harvester's list. Fetch-it cannot stop direct access either. To do what you are requesting takes other means, like adding code to your .htacess file, assuming you are on *nix, that disallows access. This is not tested, but should give you a start

Code:

RewriteEngine on


RewriteCond %{HTTP_REFERER} ^$ [OR]


RewriteCond %{HTTP_REFERER} !^http://(.*)fred-dresken.nl\.([com|net]+)/.*$ [NC]


RewriteRule .*\.([mpg|mpeg|zip|tar|exe|avi|mov|gif|jpe?g]+)$ - [NC,F]

That will require all downloads (and other often stolen extensions) to only be accessed from your site. In other words they can still directly d/l if they know the name, but they will have to be on your site.

Posted: Sat Aug 07, 2004 10:03 am

Thanks Raven,

Sorry that i posted it as a bug.. Embarassed

Is this the .htaccess in the nuke root or in the folder where the files are or both..

There is a Nuke site in the netherlands called mozeskriebel.nl and every new download I ad to my folders are within 5 min. or so also active on there site, I can see a crawler from them in my IP-Tracking but I don't know how they found the original file names so that they can access them directly.. and they do that on many sites, weird. Bang Head

But I will start with your answer.

Thanks again,

Fred Mr. Green

Posted: Sat Aug 07, 2004 10:05 am

Place that in your downloads folder.

Posted: Sat Aug 07, 2004 10:17 am

Tested it,

After fetching the file this is the result, and I have CGI access for this directory.

The server encountered an internal error or misconfiguration and was unable to complete your request. Your administrator may not have enabled CGI access for this directory.

Any idea?

Thanks anyway,

fred Mr. Green

Posted: Sat Aug 07, 2004 10:25 am

Try replacing the third line with this

Code:

RewriteCond %{HTTP_REFERER} !^http://(.*)fred-dresken.nl\.([com|net]+)/.*$ [NC]

I corrected the post above.

Posted: Sat Aug 07, 2004 10:32 am

Same result as before.. Crying or Very sad

thanks

Posted: Sat Aug 07, 2004 10:57 am

Then it would appear that your host may be restricting certain .htaccess directives. Check with them to see if they can explain. This assumes that mod_rewrite is installed. Is it? If not, that could also be the problem.

Hangin' Around Joined: Jul 21, 2004 Posts: 47

Quote:

RewriteEngine on
RewriteCond %{HTTP_REFERER} ^$ [OR]
RewriteCond %{HTTP_REFERER} !^http://(.*)fred-dresken.nl\.([com|net]+)/.*$ [NC]
RewriteRule .*\.([mpg|mpeg|zip|tar|exe|avi|mov|gif|jpe?g]+)$ - [NC,F]

Can i use this as an antileech method to protect my downloads from leeching?
And if so should i change that "fred-dresken.nl" to my sites url?
(i know i have to make an htaccess file in the downloads folder)
Dobbelina
Smile

Posted: Tue Aug 10, 2004 7:37 am

Hi Dobbelina,

It is a tricky method if you don't have CGI access as Raven told in a early post.

You probably can use Fetch-IT first and if that don't help then you can try this feature, but that only works when you have CGI access to that folder. Inform about that with your hosting provider.

Fetch-It - http://www.2thextreme.org/

Greets,

Fred Mr. Green

Posted: Tue Aug 10, 2004 9:43 am

dobbelina wrote:

Quote:

RewriteEngine on
RewriteCond %{HTTP_REFERER} ^$ [OR]
RewriteCond %{HTTP_REFERER} !^http://(.*)fred-dresken.nl\.([com|net]+)/.*$ [NC]
RewriteRule .*\.([mpg|mpeg|zip|tar|exe|avi|mov|gif|jpe?g]+)$ - [NC,F]

Can i use this as an antileech method to protect my downloads from leeching?
And if so should i change that "fred-dresken.nl" to my sites url?
(i know i have to make an htaccess file in the downloads folder)
Dobbelina
Smile

That should work. I don't thing CGI or Apache module should matter with .htaccess.

Posted: Wed Aug 11, 2004 9:06 pm

Ok,many thanks i'll give it a go Very Happy

Dobbelina