NukeSeo, Webmaster tools, bots

Posted: Sat Dec 26, 2009 10:58 am

I started looking at my bike club site to see how (and how well) it is indexed. While it has been discussed several times in the Forums and is documented in our Wiki

Only registered users can see links on this board! Get registered or login!

I still have some open questions.

First, NukeSentinel seems to block Googlebot by default. In other words, 'googlebot' is listed as one of the harvesters when you display harvesters and the default blocker settings are to block it. So, and excuse my ignorance, wouldn't google be kept out of any site with these settings? Should an admin change either the list of harvesters or turn off the harvester blocking in NS configuration? What are best practices? I don't want harvesters crawling all over my sites but I don't want to keep the "good" ones out completely.

Second, I went to use the Google Webmaster tools. They want you to insert a tag:

Code:

<meta name="google-site-verification" content="a random string" />

to verify that you are the owner of the site. I can see from the documentation in the Wiki, previous posts in the forums, and my own site how to change the meta title, description and keywords, but I don't see where to add a meta name tag. Rather than hack header.php or some such I thought I'd ask if there is a way to do this. I looked at the nukeseodh block and that just seems to tell me what my title, description and keywords are.

And just to remind anyone reading this by accident, if you've upgraded to 2.4 and you've previously modified meta.php to customize your keyords you need to go into nukeseo and redo it. Look at view source on one of your pages and see if the meta section is coming out the way you want it. [/code]

Posted: Sat Dec 26, 2009 1:59 pm

Regarding the site verification, it's been awhile since I did it, but I'm pretty sure it's just a temporary tag so once your verified you can remove it.

I use includes/custom_files/custom_head.php to include some custom stuff I want in my <HEAD>, such as js copyrights or custom meta tags such as this.

Posted: Sat Dec 26, 2009 4:55 pm

if i'm correct webmaster tools also give you the option to create a html file on your server for identification!

Posted: Sat Dec 26, 2009 8:21 pm

I'm not seeing that option Whitehat but spasticdonkey's suggestion should work. I just need to find time to play with it. I didn't have much luck a while back using custom_mainfile but the way custom_head is included in header.php should make putting in the tag I need pretty straightforward.

Still looking for feedback on the NS and best practices parts of the post though.

Posted: Sat Dec 26, 2009 8:25 pm

In the verification dialog window in webmaster tools you must see on the LEFT box/block/colum a text saying :

verify with meta-tag
verify with HTML

http://www.google.com/support/webmasters/bin/answer.py?answer=35658&hl=en

sorry i edited it must be on the LEFT side of your page ... anyway folow the link provided it explains and is much easyer then include it in the metatag

Posted: Sun Dec 27, 2009 10:45 am

Thanks Whitehat. That worked. I didn't see the html option at first but you are right that it is a lot simpler than modifying the meta data.

Incidentally, looking a bit further into NukeSentinel it appears to me that the initial (default) setting of the Harvester Blocker is "off". The harvester table itself (nsnst_harvesters) comes with a pretty extensive list of harvesters so if you turn it on in the blocker configuration you'll be blocking googlebot and a lot of other stuff. On my bike club site I must, at some point, have turned harvester blocking on. I still wonder what "best practices" are though I recognize that it could vary depending on the purpose of your site and how widely you want to advertise. Right now I've just allowed in Googlebot but I suspect that there others I should allow -- if I should be blocking harvesters at all.

Posted: Sun Dec 27, 2009 9:07 pm

Quote:

The harvester table itself (nsnst_harvesters) comes with a pretty extensive list of harvesters so if you turn it on in the blocker configuration you'll be blocking googlebot and a lot of other stuff.

I looked on one of my sites at that setting (and probably hadn't looked at it in several years) and it was set to forward Harvesters. I'm not real sure of the inner workings of this but I would have to assume the ABHarvesterList is the list of allowed Harvesters, and the blocker settings only take effect if the bot is not on that list... but just a guess. Smile

The site I was referencing does pretty well on google, and I haven't seen any crawl errors through webmaster tools either... so just my two cents.

Posted: Mon Dec 28, 2009 4:49 am

It is a list of blocked harvesters, but I am not seeing Google in my list.

Posted: Mon Dec 28, 2009 5:07 am

Googlebot is not in the list.If you don´t want use NukeSentinel for this you for any reason just add a list of harvester in your .htaccess. Harvester are bad and you should block these.

Posted: Mon Dec 28, 2009 10:50 am

Darn it if you are not right Susann. I just looked through the default list of harvesters that we load and Googlebot is not there. At some point in time in the past I must have added it (shot myself in the foot) and forgotten completely about it. So I guess the best approach is to use the default list that comes with NS and turn the harvester blocker on.