Google bots

Regular Joined: Jun 30, 2003 Posts: 81

Yesterday on my website, I had 6 vistors, all with the same ip, expect the last number was different, sam spade trace route said they where all webcrawler/google bots, how can I prevent google bots from coming on my site? or should I not worry about it?, my downloads are set to registerd users only.

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

robots.txt, for the bots that honor it, is the way to 'guide' the bots around your site. You could, of course, use .htaccess to blick the IP's, but it's a personal decision I suppose. I opt to try to guide them. Just my opinion.

Posted: Tue Sep 23, 2003 8:16 pm

Not sure I understand what you ment, do I really need the robots.txt? can I delete it from my root? or do i have to added something to it, to keep those bots off my site? like add a ban module or something?

Client Joined: Jul 18, 2003 Posts: 977

Check this address out. It helped me a lot...
Only registered users can see links on this board! Get registered or login!