|
|
Posted by Jukka K. Korpela on 02/20/07 14:25
Scripsit Cinamon Thunder:
> Google and other engines spider my website every single day and as
> result generates QUITE A BIT of traffic.
Do you mean the _spider visits_? Even if a spider visits your all pages
every day, that's still just one HEAD request per page. If that's
considerably more than normal traffic to your site, I'd be more worried
about the low usage than the spiders.
(I'm assuming that your server sends adequate Last-Modified headers so that
spiders can just send a HEAD request and see that nothing has changed,
instead of retrieving the actual content.)
> <META NAME="revisit-after" CONTENT="7 days">
Important search engines probably ignore that routinely, since there has
been too much abuse.
> Any advice would be appreciated!
Please convince me that daily visits by spiders are a _problem_. Many
webmasters would pay real money to achieve such a situation.
--
Jukka K. Korpela ("Yucca")
http://www.cs.tut.fi/~jkorpela/
Navigation:
[Reply to this message]
|