Robots.txt + Google

Client Joined: Dec 04, 2004 Posts: 757

Hello all, I have always used the default Robots.txt file and never make any changes to it, I was wondering if I should be making changes to it to help in search engines etc or does it as default do its job ok?
Cheers
mrix

Posted: Mon May 16, 2005 4:50 am

You can check your robots.txt here:
[ Only registered users can see links on this board! Get registered or login! ]

It´s a reall good robots.txt syntax checker.

Posted: Mon May 16, 2005 4:56 am

Hi I tried that test out and had one error which said
Disallow: admin.php
We advise you to start a file/directory name with a leading slash char (Example: /private.html).

dont understand it to be honest?
thanks for your help
mrix
p.s
my robots.txt looks like this
User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

actually this is the standard robots.txt file for 7.6 and 7.7? is it wrong?

Posted: Mon May 16, 2005 5:04 am

Try this for admin:

User-agent: *
Disallow: /admin.php
Disallow: /admin/

Posted: Mon May 16, 2005 5:56 am

Hello again, I have tried that test again with your changes and all is pasted fine, I cant understand though why would the makers of phpnuke would have as default the robots.txt the other way etc?
thanks again
mrix

Posted: Mon May 16, 2005 6:46 am

I don´t know. But i checked my Nuke 6.5 severall times with different robots.text checkers before and this one is the best of all. There are also no errors, if u use crawl-delay for some bots.

Worker Joined: Oct 07, 2003 Posts: 211

For some good info on robots.txt, look at this one:
[ Only registered users can see links on this board! Get registered or login! ]

Also, here is a really good set of tutorials:
[ Only registered users can see links on this board! Get registered or login! ]