How many times does googlebot visit your site?

Posted: Sat Sep 11, 2004 12:24 pm

Raven, can you tell me how many times googlebot visits your site on a daily basis?

It should be around the 200+ mark but I don't see your forum posts becoming visible in google yet.

Just a rough figure will do.

Posted: Sat Sep 11, 2004 12:35 pm

I just checked your listings in google and hardly anything has changed listings wise since you implemented dynamic titles.

Have you accidentally blocked googlebots from spidering your site?

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

Not to my knowledge. Do you want me to send you my robots.txt file? I sent it to you once but I will if you want.

Posted: Sat Sep 11, 2004 12:46 pm

Have you got an easy way of tracking visits?

I use msanalysis http://www.storebuilder.co.uk/analysis-s10-o1-.html

So you can see google visited 600+ times today.

There is something definitely amiss with your google listings. I think you suspected that?

Posted: Sat Sep 11, 2004 12:49 pm

Yes but I don't know what it is. I only show 104! What IP range does it use?

Posted: Sat Sep 11, 2004 1:05 pm

I made a change to my robots.txt. Can you email me yours?

Posted: Sat Sep 11, 2004 1:12 pm

Crawler 7 is 64.68.81.148 this is a new crawler that only turned up recently.

the main crawlers that you are missing are the freshbot ones 64.68.62.* - I think somehere, and probably not in your robots.txt but maybe in sentinel or your.htaccess you are denying them access.

I have been looking everyday for your post to turn up "where is audioslaved", not only because we want to know where he is but also because it's a measure of how your site is being indexed.

No one else (I would think) has the phrase "where is audioslaved" in their page title so once you see your site appear no 1 in google for this we are heading in the right direction.

RavensScripts

Posted: Sat Sep 11, 2004 1:17 pm

Mine is vanilla

User-agent: *
Disallow: admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

Posted: Sat Sep 11, 2004 1:18 pm

Well, I'm not blocking the IP anywhere, so I'll just modify my robots.txt and see what happens.

Client Joined: Apr 10, 2004 Posts: 649 Location: UK

Did you put this code in your robots.txt file?

Quote:

User-agent: Mediapartners-Google*

before the disallows?

That helps.

Apologies if I'm teaching granny how to suck eggs here.
Embarassed

Posted: Sat Sep 11, 2004 1:20 pm

Yep

Posted: Sat Sep 11, 2004 1:22 pm

I am blocking these headers:
DELETE
SEARCH
TRACE

Do you know if Google uses these?

Posted: Sat Sep 11, 2004 1:28 pm

Can you check your logs to see if 64.68.62.* is accessing your site.

Posted: Sat Sep 11, 2004 1:30 pm

I'm sending you an email!

Posted: Sat Sep 11, 2004 1:39 pm

I don't know how you do this?

I am blocking these headers:
DELETE
SEARCH
TRACE

Blocking these headers where?

Posted: Sat Sep 11, 2004 3:06 pm

In NukeSentinel. Do you know if Google uses the Search?

Posted: Sat Sep 11, 2004 4:48 pm

storebuilder wrote:

Have you got an easy way of tracking visits?

I use msanalysis http://www.storebuilder.co.uk/analysis-s10-o1-.html

So you can see google visited 600+ times today.

There is something definitely amiss with your google listings. I think you suspected that?

Your site is unreachable right now.

Edit: Must be the same problem(s) you've been having with your host - it's up and down.

Posted: Sun Sep 12, 2004 9:26 am

Oh I see. No. I don't know.

Posted: Sun Sep 12, 2004 10:52 am

I have been reading all morning about 'googlebot no longer visits my site' and the general concensus is that 'who knows why'? It's very frustrating as it used to hammer my site and now is alomost never here. I cannot thing of anything that I have done or haven't done.

Posted: Sun Sep 12, 2004 5:05 pm

I think if the content stays basically the same which it does here, as it's a nuke and sentinel support site that once they know that that is the content they no longer need to visit so often.

It's not like a news site where the content is changing daily or weekly or even monthly.

Just a guess. I mean if you posted an article up about a different topic like naturism or clockwork rabbits they'd notice it was new content and pay the site more visits again.

(hope ya dont mind me butting in)

Posted: Mon Sep 13, 2004 1:09 pm

Nice guess muffin, but wide of the mark in this case.

One of the reasons for rewriting the url's so that they appear as HTML is that they carry more importance than database driven pages.

If you were a search engine and someone had 2,000 html pages which were updated regularly then you would figure that someone is putting a lot of effort into that site and so it must be important.

In any case there is a considerable amount of activity in the forums here and as google loves topical content it should be indexing it immediately.

To highlight what the problem is with this site then look at this post http://www.storebuilder.co.uk/postt215.html and spot the datestamp and then do this search http://www.google.co.uk/search?hl=en&ie=UTF-8&q=website+placement+uk&meta= - comparatively my forums are a graveyard as I'm still building traffic in a very competitive field.

By combining googletap with a page title mod you are able to get new content listed in a search engine friendly way very quickly.

The funny thing is that you are getting 1,000 visits from msnbot daily and I get none whereas I get 1,000 from google and none from msn - so there's another missing piece in the jigsaw.

I've heard of other people who have experienced googlebot drying up on them so I'm off to see the wizard...

Posted: Mon Sep 13, 2004 1:31 pm

This is a guess - you are being penalised for duplicate content.

If you do a site:ravenphpscripts.com then the return is a list of pages with no descriptions. Just an option for "similar pages".

Because you have had the same page title for every single page then it's been considered duplicate content and so the googlebot's have dried up.

I found one post indexed since you did the dynamic titles mod and so here's an example search http://www.google.com/search?hl=en&lr=&ie=UTF-8&q=pc+killer+to+use%3F to bring it up.

Maybe at this point it's best to have some patience and see if the bots start coming back to index your "new" content.

Posted: Mon Sep 13, 2004 1:36 pm

Thanks Tony and let me know if the 'Wizard' comes up with anything!

Posted: Mon Sep 13, 2004 1:50 pm

Oh he's good, he's really good Wink

registered · Posted: Mon Sep 13, 2004 3:05 pm

I had some links in my downloads and weblinks which had parenthesis in them -- I changed these to brackets after Sentinel was installed when I realized people hitting those pages where getting banned. Unfortunately, for some reason, a googlebot has been banned before as a result of one of those pages, and it's still cached in Google's engine. Twice since, I have had to remove a googlebot from my banned list for accessing the cached page......was really weird, and the last time I only noticed this when I realized google wasn't spidering my site as much as it used to......