Posted by Brian Cryer on 02/20/07 09:40

"Cinamon Thunder" <k@postman.co.za> wrote in message
news:ered3g$2u8$1@wblv-ip-nnrp-1.saix.net...
> Google and other engines spider my website every single day and as result
> generates QUITE A BIT of traffic.
>
> I've tried:
>
> <META NAME="revisit-after" CONTENT="7 days">

Personally, I don't bother with this one. Its a request for robots to
revisit after 7 days, but its only a request. It doesn't mean only revisit
after 7 days. Also, its a hint for the current page. So you are saying for
*this* page *please* visit again after 7 days.

> --but the engines still visit daily.
>
> I dont want to use:
>
> <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
>
> --as it will prevent spidering all together unless I remove the tag every
> week or so for a few days?
>
> I'm not sure about robot.txt as I want the entire site indexed (just not
> every single day) and no pages excluded.

How many pages to you have? In my experience search engines don't tend to
pull down the same pages regularly (unless they've changed), but if you have
a large site then they can spend days (or more) working through all the
pages on your site.

Regarding google, whilst I don't know a way of saying only spider one day in
seven, you can change the crawl rate. Go to
https://www.google.com/webmasters/sitemaps/ (which redirects you), sign up
for a Google Webmaster tools account. There is a minor hoop to jump through
so they they can validate that you are responsible for the domain you claim
to be responsible for, but thereafter you can change the crawl rate down
from "Normal" to "Slower".

Be aware that this will mean it takes longer for google to pick up changes
and new pages on your sit.e

Can't advise about other robots. Normally about all you can do is block
them, which you probably don't want.

Hope this helps.
--
Brian Cryer
www.cryer.co.uk/brian

[Back to original message]