Robot Txt

Worker Joined: Oct 26, 2004 Posts: 134 Location: Boston

I am getting a lot of bots not complying to the robot txt but did come across this code that i would like to use but need to know 2 things what do i do here

Quote:

Replace the broken pipe(¦) with a solid one in .htacces snippet

And what would be the right path to the .htaccess file ist it something like this /home/joe/public_html/.htaccess Thanks For the help with this

Notes:

Add the robots.txt snippet days before luring bots to the trap. This gives the good bots time to read the disallow and obey.

chmod .htaccess to 666 and getout.php to 755(please correct me here if I'm wrong)

Replace the broken pipe(¦) with a solid one in .htacces snippet.

Edit getout.php with the real path to your .htaccess file and also change the email to your own so you will recieve the "spider alert".

Robots.txt

User-agent: *
Disallow: /getout.php

Code:

.htaccess(keep this code at the top of the file) 





SetEnvIf Request_URI "^(/403.*\.htm¦/robots\.txt)$" allowsome 


<Files *> 


order deny,allow 


deny from env=getout 


allow from env=allowsome 


</Files>

getout.php

Code:

<?php 


$filename = "/var/www/html/.htaccess"; 


$content = "SetEnvIf Remote_Addr ^".str_replace(".","\.",$_SERVER["REMOTE_ADDR"])."$ getout\r\n"; 


$handle = fopen($filename, 'r'); 


$content .= fread($handle,filesize($filename)); 


fclose($handle); 


$handle = fopen($filename, 'w+'); 


fwrite($handle, $content,strlen($content)); 


fclose($handle); 


mail("me@mysite.com", 


"Spider Alert!", 


"The following ip just got banned because it accessed the spider trap.\r\n\r\n".$_SERVER["REMOTE_ADDR"]."\r\n".$_SERVER["HTTP_USER_AGENT"]."\r\n".$_SERVER["HTTP_REFERER"] 


,"FROM: trap@mysite.com"); 


print "Goodbye!"; 


?>

registered · Site Admin Joined: Jun 04, 2004 Posts: 6437

There are several efforts to trap so-called "bad bots" - those that do not respect the robots.txt directives. NukeSentinel has the ability to block known bad bots, and there has been other discussion on this site about alternatives.