robots.txt

Worker Joined: Dec 23, 2005 Posts: 226 Location: UK

First of all Well Done to Raven and the team for all the hard work dragging phpnuke into the 21st century.

Just a couple of questions regarding robots.txt

What is Crawl-delay: 5 ?
and should ShortLinks be included?

Posted: Mon Mar 12, 2007 7:20 am

Crawl delay should slow down the crawling of bots on your site. Although I do recall reading somewhere that Google ignores this option, but if you have your site registered with Google you can specify a delay there instead.

I'm at work at the moment so I do have any files to look at, but I guess bots don't necessarily need to crawl the shortlinks directory so you could include if you wanted to. Maybe Montego can confirm/clarify this?

I don't think is really a bug as such, so I have moved the post.

Jakec

registered · Site Admin Joined: Jun 04, 2004 Posts: 6437

They should not need to crawl the shortlinks directory, and it would be better if this is added to the robots.txt file.

registered · Posted: Tue Mar 13, 2007 6:22 am

I agree. My oversight. Sorry about that.

Posted: Tue Mar 13, 2007 6:32 am

Not a big deal. I don't think it will really hurt anything.

Posted: Tue Mar 20, 2007 11:57 am

Did we ever added Audioslaves Google Tap folder into the robots.txt ?
There isn´t any instruction where ist says you should add this folder to the robots.txt.
In my opionion its not required because I never saw a bot in the past years indexing the Google Tap folder from my website.

Posted: Wed Mar 21, 2007 5:38 am

Do not know Susann. Just thought that it could not hurt is all.

Involved Joined: Apr 05, 2006 Posts: 263

i believe you have to enter this to stop googlebot indexing your site

in the robots file at the /Root

User-agent: Googlebot
Disallow: /

Posted: Wed Apr 04, 2007 12:47 pm

Why would you want to stop Googlebot from indexing your site, unless you never want it appear in the Google search? Confused

Posted: Wed Apr 04, 2007 12:55 pm

jakec wrote

Quote:

Although I do recall reading somewhere that Google ignores this option

in reply to first post

so thought i would add that you need this to stop googlebot

someone might want it. never know

Posted: Wed Apr 04, 2007 2:06 pm

OK I see what you are saying.

My point was that although Googlebot doesn't recognise the crawl delay in the robots.txt files, you can apply a delay if your site is registered with Google.

If you block Googlebot it is unlikely that people will find your website. Maybe that is what you want, if it is a private site.

Posted: Wed Apr 04, 2007 5:26 pm

i have seen some guys /ladies on here have private sites (friends/ family)so it would be usefull to them.

i can see if you have a really busy site it would also be very usfull to with other indexing robots or what not on the net.

i do think its usefull as the google site i seen, on googlebot it has more than 10,000,000 robots out there.

Posted: Thu Apr 05, 2007 2:44 am

Like I said, I understand what you are saying, and I too have some personal sites that I don't really want indexed, but also I'm not bothered if people did visit them. So blocking good bots in this way is one solution.

...but the original question was:

hinksta wrote:

What is Crawl-delay: 5 ?
and should ShortLinks be included?

Posted: Thu Apr 05, 2007 2:06 pm

Going off topic slightly but if you want to stop all bots (that adhere to the robots.txt) then you can simply use

Code:




User-agent: * 


Disallow: /

As far as I am aware, the Crawl-delay only works with Slurp (at least Yahoo claim its a directive specifically for their bots).
http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html