How-To combine DL's leeching prevention & Surveillance?

Posted: Wed Oct 06, 2004 3:01 am

How to safely combine *downloads leeching prevention* and *surveillance to it*?

[BASICS]

This issue/subject has been passed in multiple occasions, but hopefully this will bring out something to those of us who are now actively figuring out the preferable selection/utilization approaches.

Quote:

::: TOP-2 Reasons for applying this practise :::

- ensuring that only up-to-date information[entities] are accessesed[downloaded] from specific folders
- allocating an efficient tweak for *hotlinking* prevention (with associated lightweight email based notifications)

In the next, we're going to pass through (2) pcs of extremely closely related approaches used optionally to achieve the above referred target - with combined *leeching prevention* and *associated lightweight surveillance* to it (extracting information out of the point of origins performing the leeching).

Quote:

1) leeching prevention

The referred (2) pcs of CASE approaches are a follows:

::: CASE#1 :::

* [allowed] : "blank referrer" (as a referrer)
* [allowed] : "own site" (as a referrer)
* [allowed] : "trusted site" (as a referrer)
* [disallowed] : "any other site" (as a referrer)

::: CASE#2 :::

* [disallowed] : "blank referrer" (as a referrer)
* [allowed] : "own site" (as a referrer)
* [allowed] : "trusted site" (as a referrer)
* [disallowed] : "any other site" (as a referrer)

Quote:

2) associated surveillance

Regardless of the (2) optional *alternatives* chosen, a BLOCKED USER gets redirected to an allocated subdomain containing INFO-page (as seen in the undebeneath DEMO section), which, as accessed, will send out a pre-formulated email to Site Admin for *email based records* archiving purposes. (there's an associated example attached to the end of this thread)

"Hey, hold on a moment!"- what's the difference between the two *leeching prevention* approaches above? Those look like identical to me?!

Yes, indeed - almost, but not quite. There may, or may not, be severe consequencies from Site Administrator's point of view dependending on your Status (Business/non-Business, web presence coverage targets, etc.) and the approach that you are about to select (or have selected) as a backbone *leeching prevention* solution.

Which brings us into the *core issue* through a shortcut - whether or not to allow *blank referrer* equipped visitors to access your downloads folder? This may apparently appear as a case, which could have a quick answer to it, but I do believe that it is justified to claim that both approaches have their pros and cons, and thefore are worth a small journey to pass through.

So, whenever we are talking about the *BROWSER ASSOCIATED* $HTTP_REFERRER information, we should remind yourselves that it is closely (if not directly) related to the present STATE OF THE USED WEB BROWSER, PRESENCE OF PROXIES/FIREWALLS, that are either used or not when accesing INTERNET.

State of the used browser? C'mon - you must be kiddin'? Nope, being serious - as soon as you have initially opened your browser as an application at your PC, the first instance (web server) that is met by it, will identify your BROWSER as a *blank referrer* instance/visitor. And why's that? That is simply due to a fact, that there's no possible (natural) source for "getting tagged" by the $HTTP_REFERER information, as the target site has been #1 accessed, just after the occurred launch up at our PC.

Should I get worried from Downloads protection point of view? You should be cautious at all times - however, I find it very difficult to believe that anyone would consistently rely on downloading/leeching your KNOWN FOLDERS, based only onto a fact that your proctection scheme would be based on *blank referrer ALLOWANCE* - things may not be that black'n'white at all, though. There is a certain amount of solid reasons to believe that no *fool proof* DL security solution can rely solely upon *BROWSER ASSOCIATED* *blank referrer* detection and DISALLOWANCE - unless it is specifically required.

Furthermore, as we realize that the *browser associated* $HTTP_REFERER information has those mentioned characteristics being equipped with, we might be soon asking ourselves; "ok, so be it - is there anything else?" Actually, there is. Now, that we have scratched the surface related to use of *blank referrer* information in protective sense, we should next turn down the tables and look for the potentially present downside(s) from *DISALLOWANCE* point of view. And here's the deal - should you persist to use the *unreliable* blank referrer* DISALLOWANCE as a part of your Downloads protection scene's perspective, you'd be most probably soon facing the beasts of your own actions - ones, that in worst case would bring down your credibility as a Services provider - i.e. customer support calls due no other reasons than people getting blocked due to applied *blank referer* DISALLOWANCE policy.

So, do you mean that it's the use of active *blank referrer* blocking that would cause this? Exactly, as in many global regions, there are ISP's present which policy covers DISALLOWANCE for sending the *BROWSER ASSOCIATED* $HTTP_REFERER information - there are also several people accessing through firewalls/proxies without the *BROWSER ASSOCIACTED* $HTTP_REFERER information being sent along. So, given a thought of the groups of people being left out in the cold, unable to access your *blank referer DISALLOWANCE* backed security approach, it would probably get chills out of you.

So, should you utilize .htaccess file based $HTTP_REFERRER procetection with *blank referrer* DISALLOWANCE, please think again before choosing to block *blank referrer* information equipped visitors.

What possible general consequencies could all that above have then? Well, in real life, as you are surrounded by security experts, I'd say, possibly no true threads, as it is commonly understood that relying solely on *BROWSER ASSOCIATED* $HTTP_REFERER information identification/evaluation will bring you down eventually, so to speak.

So, let's take a look at the EXAMPLE scene - protecting your *virtual* downloads folder - what is it that actually happens differently between these (2) approaches when applying those at separate instances? (you will be given a chance to safely test both approaches in the next - a real life DEMO-implementation)

[please note that there are other security measures present as well (as opposed to .htaccess) - this is just to take peak at the .htaccess file based protection - and particularly, to a single example approach which should be by no means considered to cover it all.]

###########
# CASE #1 # (RECOMMENDED)
###########

[.htaccess file example - to be placed into the *downloads* folder of interest with modified preferences]

Quote:

###########################################################################
# with *NO* blank referrer protection* (this is the recommended approach) #
###########################################################################

RewriteEngine on
# allowing visitor with *blank referer* information to access the folder/items
# allowing mother site access - either from *long* or *short* form of URL
# allowing mother site's subdomain access - either from *long* or *short* form of URL
# allowing TRUSTED site access - either from *long* or *short* form of URL
# defining the set of *guarded item* suffixes. declaring the forward target due to set restricitions.
#
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?ec-clan.org(/)?.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?bf1942live.ec-clan.org(/)?.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?/ravenwebhosting.com(/)?.*$ [NC]
RewriteRule .*\.(txt|tar|gz|rar|zip|mpg|mpeg|mp3|exe|avi|mov|swf|gif|jpeg|png)$ http://blocked.ec-clan.org [R,NC]

Quote:

::: .HTACCESS FILE INTERPRETATION :: (the above .htaccess file struct)

VISITOR *BROWSER* CONDITIONS - [browser equipped with $HTTP_REFERER information]

::: CONSEQUENCIES :::

* [allowed] : "blank referrer" (as a referrer)
* [allowed] : "own site" (as a referrer)
* [allowed] : "trusted site" (as a referrer)
* [disallowed] : "any other site" (as a referrer)

If the visitor is evaluated as disallowed, the used browser will get redirected to *TARGET PAGE*

Quote:

::: TEST SECTION [with *NO* blank referrer protection*] :::

TEST#1 : Access the following *DOWNLOAD LINK* : Only registered users can see links on this board! Get registered or login!

RESULT#1 : (FORBIDDEN: you were redirected to INFO-page and given a noticce that this file cannot be leached)

TEST#2a : Now, please open up a new BROWSER WINDOW
TEST#2b : Copy/Paste the above (5 lines up) *DOWNLOAD LINK* to your browser's URL field - then hit <ENTER>

RESULT#2 : (SUCCESS!!: you managed to download it, as there INTENTIONALLY WAS NO *blank referer* protection being used)

RECOMMENDED - please see the argumentation/introduction above

###########
# CASE #2 (UNRECOMMENDED)
###########

[.htaccess file example - to be placed into the *downloads* folder of interest with modified site preferences]

Quote:

####################################################################
# *with blank referrer protection* (this is UNRECOMMENDED approach) #
####################################################################

RewriteEngine on
# disallowing visitor with *blank referer* information to access the folder/items
# (as there's no statement to it)
# allowing mother site access - either from *long* or *short* form of URL
# allowing mother site's subdomain access - either from *long* or *short* form of URL
# allowing TRUSTED site access - either from *long* or *short* form of URL
# defining the set of *guarded item* suffixes. declaring the forward target due to set restricitions.
#
RewriteCond %{HTTP_REFERER} !^http://(www\.)?ec-clan.org(/)?.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?bf1942live.ec-clan.org(/)?.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?/ravenwebhosting.com(/)?.*$ [NC]
RewriteRule .*\.(txt|tar|gz|rar|zip|mpg|mpeg|mp3|exe|avi|mov|swf|gif|jpeg|png)$ http://blocked.ec-clan.org [R,NC]

Quote:

::: .HTACCESS FILE INTERPRETATION :: (the above file struct)

VISITOR *BROWSER* CONDITIONS - [browser equipped with $HTTP_REFERER information]

::: CONSEQUENCIES :::

* [disallowed] : "blank referrer" (as a referrer)
* [allowed] : "own site" (as a referrer)
* [allowed] : "trusted site" (as a referrer)
* [disallowed] : "any other site" (as a referrer)

If the visitor is evaluated as disallowed, the used browser will get redirected to *TARGET PAGE*

Quote:

::: TEST SECTION [with *NO* blank referrer protection*] :::

TEST#1 : Access the following *DOWNLOAD LINK* : Only registered users can see links on this board! Get registered or login!

RESULT#1 : (FORBIDDEN: you were redirected to INFO-page and given a notice that this file cannot be leached)

TEST#2a : Now, please open up a new BROWSER WINDOW
TEST#2b : Copy/Paste the above (5 lines up) *DOWNLOAD LINK* to your browser's URL field - then hit <ENTER>

RESULT#2 : (FORBIDDEN: you were redirected to INFO-page and given a notice that this file cannot be leached,
as there's *blank referrer protection* being INTENTIONALLY SET present)

[UNRECOMMENDED - please see the argumentation/introduction above]

So, that's bout the DEMO scene - hopefully those passed things shed some light to the *blank referrer* issues and to how the different scenes are mapped "out to"/"in between" each other.

::: ASSOCIATED LIGHTWEIGHT EMAIL BASED NOTIFICATIONS :::

So, as we noticed above, each time an access was denied, the Test Person was directed to the .htaccess file declared INFO-page. This page is used for gathering light-weight email based reports on each access that will get to hit to the page. Should you find it useful from your point of view, please find in the next an CODE INSERTION SECTION that should be placed as a default just after the PHP opening TAG (you should still pay to those opening/closings tags)

You'll find the very *email routine code snippet* being attached into an entity which is accessed via DEMO CASE#1 herein - with the overall struct of the INFO-page...

Quote:

::: CODE INSERTION QUICKREF :::
* just insert the snippet into the beginning of a PHP file
* do note the correct use PHP opening and closing tags

|----> beginning of code insertion

Quote:

<?php
$acceptemail=1; // set to "1" to receive mail, to "0" not to receive
$to='spam@ec-clan.org'; // recipient email address - needed
$blocksubject="NOTE: INFO from BLOCKED.EC-CLAN.ORG"; // change the subject as seen appropriate
// beginning of email routine
$ip=getenv("REMOTE_ADDR");
$date=date("m/d/Y H:i:s");
if($acceptemail==1) {
mail($to,$blocksubject,"\n
----------------------------
- DOWNLOADS LEECHING ALERT -
----------------------------
ABUSE DATE&TIME : $date
IP ADDRESS : $ip
HTTP HOST : $HTTP_HOST
HTTP REFERER : $HTTP_REFERER
FROM IP ADDRESS : $SERVER_ADDR
ALERTING SRC FILE : $SCRIPT_FILENAME");

}
// end of email routine
?>

|----> end of code insertion

::: BROWSER ASSOCIATED *BLANK REFERER* SUMMARY :::

Quote:

SITE VISITOR *BROWSER* CONDITIONS - [browsers NOT equipped with $HTTP_REFERER information]

Possible reasons for visitor *BRWOSERS* not being equipped with $HTTP_REFERER information
* (1) accesing via proxy/firewall ( => visitor considered as *blank referer*)
* (2) accesing through browser that has just been opened ( => visitor considered as *blank referer*)
* (3) accesing through browser that not allows $HTTP_REFERER informatioon to be sent ( => visitor considered as *blank referer*)
* (3) accesing via ISP that not allows $HTTP_REFERER information to be sent ( => visitor considered as *blank referer*)

As you manage to gather information on the leeching attempts (and parties involved) - based on the INFO-page emailed extract - you may be then more prepared to perform the some of the available actions needed to protect yourself and your DL sections in the future.

All in all, given a different point of view - what might be comprehended as *unprotecting oneself* in the first place, might be afterall be *protecting oneself* in a considered manner.

Hope this helps someone.

BR,

-beetraham

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

Most Excellent!!

registered · Posted: Sun Dec 05, 2004 8:41 pm

I'm reading through all that and am getting a headach. :ROTFL

I want to block my downloads folder from being linked from anywhere. What exactly do I need to do to allow only www.phenylshouse.com and nothing else?

I wan't an email to be sent to me when someone tries to link to any of my downloads. I'm not sure what to name that php file for the email thing. Or does it go in the htaccess as well?

Posted: Sun Dec 05, 2004 8:42 pm

Ok, I just had a revelation. Is blank referrer mean someone who just types in an address in their browser? Cuz I don't really want that either. I want people to get downloads from my download area. Very Happy

Posted: Mon Dec 06, 2004 1:08 am

Thanks for the inquiry.

TheosEleos wrote:

Ok, I just had a revelation. Is blank referrer mean someone who just types in an address in their browser?

OK, so to put it in a nutshell; in this case a "blank referer" is a "user browser web page access" being considered NOT to have any HTTP_REFERER information being sent along with the browser access.

From that perspective, there may be several reasons for HTTP_REFERER information not being propagated as opposed to situations in which it is experieced to be present as a info source.

This referred "lack of information" may be related to at least the following point of origins;

* user firewall policy (information stripped off by the FW)
* browser policy (information stripped off by the web browser)
* ISP policy (information stripped off by the user's ISP)
* non-referrer page initialized access (the target page is accessed as 1st page during the web browsing session)
* "reloading the page again"
* "some other source"

TheosEleos wrote:

...Cuz I don't really want that either. I want people to get downloads from my download area. Very Happy

I'd definately want to emphasize the disencouraged use of BLANK REFERRER orienteed DL protection - that's exactly the reason why it has been suggestively tagged as an "unrecommended alternative/approach" in the first place.

Hope this will shed some light into the big picture.

Thanks a lot,

-beetraham

Posted: Mon Dec 06, 2004 1:50 am

TheosEleos wrote:

I want to block my downloads folder from being linked from anywhere. What exactly do I need to do to allow only www.phenylshouse.com and nothing else?

IMHO, HTTP_REFERER based approach does NOT provide a webmaster a solution with 100% controllability over the "source of referrer page origin investigations" - one simply cannot trust the results due to nature of the user browser provided HTTP_REFERER information.

You may want to consider an alternative SESSIONS based approach as an approach having the MOST EXACT controllability over the DL-section accesses - as that one is not being discussed in this thread, I'll bypass the furhter discussion related to it herein.

TheosEleos wrote:

I wan't an email to be sent to me when someone tries to link to any of my downloads. I'm not sure what to name that php file for the email thing. Or does it go in the htaccess as well?

OK, great - a good question. In that particular case, I'd recommend you to proceed as follows:

(1) define the "redirection target page" in DL-folder specific htaccess file (please see the example underneath)

Code:




########################################################################### 


# with *NO* blank referrer protection* (this is the recommended approach) # 


########################################################################### 





RewriteEngine on 


# allowing visitor with *blank referer* information to access the folder/items 


# allowing mother site access - either from *long* or *short* form of URL 


# allowing mother site's subdomain access - either from *long* or *short* form of URL 


# allowing TRUSTED site access - either from *long* or *short* form of URL 


# defining the set of *guarded item* suffixes. declaring the forward target due to set restricitions. 


# 


RewriteCond %{HTTP_REFERER} !^$ 


RewriteCond %{HTTP_REFERER} !^http://(www\.)?ec-clan.org(/)?.*$ [NC] 


RewriteCond %{HTTP_REFERER} !^http://(www\.)?bf1942live.ec-clan.org(/)?.*$ [NC] 


RewriteCond %{HTTP_REFERER} !^http://(www\.)?/ravenwebhosting.com(/)?.*$ [NC] 


RewriteRule .*\.(txt|tar|gz|rar|zip|mpg|mpeg|mp3|exe|avi|mov|swf|gif|jpeg|png)$ http://blocked.ec-clan.org [R,NC]

(2) As a second task, please bind the PHP code architecture being used to deliver EMAILED notifications for you in case of TARGET PAGE accesses - naturally, you may freely choose whatever the page you desire.

I personally have considered it "semi-professionally and aesthetically pleasing" to create a dedicated domain for BLOCKED VIOLATIONS and performed actions related to those - using the very same sub-domain for Nuke-Sentinel redirection purposes as well - though with UNIQUE target pages in those cases .

In the above case EXAMPLE, a detected access violation is being redirected into a SUB-DOMAIN [http://blocked.ec-clan.org] - as the only index page in that sub-domain is INDEX.PHP, a sequence of procedures is carried out as the page is being accessed.

In order to take command over the EMAILED notifications; in this case, it is the INDEX.PHP that will contain either;

Quote:

* flat CODE insert - the PHP code related to "email routine code snippet..."
(or optionally)
* include statement grabbing an external file containing a PHP code related to "email routine code snippet..."

...being used for delivering the SUB-DOMAIN ACCESS (redirection target page) notifications.

Hope this will clear things up.

Thanks,

-beetraham

Posted: Sun Jan 30, 2005 1:35 pm

So how are SESSIONS controlled within an htaccess file?

Also, I tried to use that PHP code to e-mail any violators and it didn't work.

Posted: Mon Jan 31, 2005 3:32 am

jonmcc33 wrote:

So how are SESSIONS controlled within an htaccess file?

Also, I tried to use that PHP code to e-mail any violators and it didn't work.

Hi,

Thanks for the notices, hopefully I can find some resolution upon those in the next.

Quote:

So how are SESSIONS controlled within an htaccess file?

In this "How-To" htaccess files, there is none.

Additionally, there is none being discussed in order to focus on the *web browser deliverables, ie. $HTTP_REFERER only*. In other words, the SESSIONS are simply considered as out of scope to keep it simple.

However, as you have implied, the SESSIONS are very powerful available method, when dealing with *server side deliverarables*, so to speak. Leaning on SESSIONS in protective sense would typically include examining SERVER SIDE deliverables prior to entering, say, into a page containing protected items, or a downloadable link (item) as an example.

<=> Riding on a wave of net, as an example target approach => "if you came in this server, from an instance "X" in this server, then as a webserver, I trust it's OK for you to..."

The SESSIONS may well have been discussed throughly here/elsewhere in the forums - if not, I'm positive that there are many around willing to share their thoughts from downloads protection point of view as well. Specifically, when - and when not - felt worth consideration as an implementation option.

::: SUMMARY :::

As a re-summary; what we are trying to accomplish in this "How-To", is simply to prevent *DIRECT FILE LINKAGE* into a *known file in a known downloads folder* (-> say, a file absolute path is revealed by a web site harvester targetting to hotlink the victim site DL items) using the extracted HTTP_REFERER information as a primary guard (in this occasion).

In this "How-To", we'll only examine the browser deliverables - i.e HTTP_REFERER information.

<=> Riding on a wave of net, as an example target approach => "if you browser "A" and/or firewall "B" and/or ISP "C" all allow you to deliver the HTTP_REFERER information, then I'll treat you according to rules being set in the DL folder .htaccess file directives. Otherwise, I'll treat you as a visitor being equipped with "blank referer information", for whom we'll also apply rules according to DL folder .htaccess file content."

So herein we're not relying on "SESSIONS", but more like to "BROWSER DELIVERABLES", i.e. "HTTP_REFERER" information. The HTTP_REFERER based approach should be considered as a complementary layered approach with limited coverage (SESSIONS usage not discussed in examples). The limited nature of HTTP_REFERER browser deliverable been emphasized in many occasions, also earlier in this topic.

Quote:

Also, I tried to use that PHP code to e-mail any violators and it didn't work.

The earlier given examples should cover the necessarily needed actions to make it happen. However, should I have failed to make it clear and readable enough, please find an attempt to clarify it in the next:

(1) - define the "redirection target page" in DL-folder specific htaccess file (example)

Code:




###########################################################################


# with *NO* blank referrer protection* (this is the recommended approach) #


###########################################################################





RewriteEngine on


# allowing visitor with *blank referer* information to access the folder/items


# allowing mother site access - either from *long* or *short* form of URL


# allowing mother site's subdomain access - either from *long* or *short* form of URL


# allowing TRUSTED site access - either from *long* or *short* form of URL


# defining the set of *guarded item* suffixes. declaring the forward target due to set restrictions.


#


RewriteCond %{HTTP_REFERER} !^$


RewriteCond %{HTTP_REFERER} !^http://(www\.)?ec-clan.org(/)?.*$ [NC]


RewriteCond %{HTTP_REFERER} !^http://(www\.)?bf1942live.ec-clan.org(/)?.*$ [NC]


RewriteCond %{HTTP_REFERER} !^http://(www\.)?/ravenwebhosting.com(/)?.*$ [NC]


RewriteRule .*\.(txt|tar|gz|rar|zip|mpg|mpeg|mp3|exe|avi|mov|swf|gif|jpeg|png)$ http://blocked.ec-clan.org [R,NC]

(2) - modify the DL folder located and dedicated .htaccess according to your *allowed sites specs*

(3) - insert the code snippet into target page <=> [ in this example case: http://blocked.ec-clan.org/index.php ]

Note: the above page will be associated as a "RewriteRule Target Page", please see the rule above for subdomain redirection)

(4) - testdrive the VALIDITY of your implementation by placing direct linkages in direct domains/subdomains.

Note: Please see the above example .htaccess file COMMENTS for protection coverage.

(5) - should your testdrive fail, please re-examine your implementation flow.

Thanks,

-beetraham

Posted: Mon Jan 31, 2005 8:45 am

It does work in denying people access. I have a couple PHP files that a friend gave me that do the reporting instead when I log into it.

I put the e-mail script into it and it just doesn't e-mail at all.

Posted: Mon Jan 31, 2005 3:40 pm

Quote:

It does work in denying people access. I have a couple PHP files that a friend gave me that do the reporting instead when I log into it.

OK, that's good to hear.

Quote:

I put the e-mail script into it and it just doesn't e-mail at all.

I also double-checked the email code snippet by isolated code execution approach today, and it works perfectly with my Linux Apache. So, when it comes to email code snippet, that's OK.

Your experiences may, or may not, have got something to do with your PHP - or in the simplest case, with your code instantiation. I Would need to receive the code instatiation (and associated .htaccess) prior to making any further sophisticated guesses.

It's up to you whether you want have any further resolution on that. If so, please let me know.

BR,

beetraham

Posted: Mon Jan 31, 2005 3:43 pm

[this post was an accidental clone - please ignore this]