The robots.txt file question

Client Joined: Jul 18, 2003 Posts: 977

Why is it that when I have a wildcard in the user agent and one of the disallows is set at /modules/ I still have bots in the forums? Aren't the forums covered by the /modules/ if so why are they still in there and more importantly how do I get them out of the forums? I would like them to be on the rest of the site but not in the forums. Is there someplace I have to activate the robots.txt file? Thanks.

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

bots are not required to honor any of those directives. bots either choose to or do not.

Posted: Fri Sep 19, 2003 9:13 am

Well the one I am specifically dealing with is the Googlebot. At any one time I have about 20 or so of them in my forums. I believe they maybe contributing to my max_resource problem that I have with ipowerweb. I think you remember it... (When my current contract runs out I will look at hosting with you Raven Wink

) Anyway on the google website they say their bots honor the robots.txt file but it may take awhile... so after the standard searching I should have done first, I at least found an answer to the problem... now I will see if the do in deed honor it... Thanks again Raven....