Posted by Nikita the Spider on 10/10/76 11:56

In article <1156325769.264557.183990@m73g2000cwd.googlegroups.com>,
"Paul" <desotuatail@aol.com> wrote:

> I am tearing my hear out. It apears my website is under atack from
> these search engins. I have heard that I can place code in my header
> som where to stop this. Any help/
>
> the browser information that I have collected show up the following
>
> Mozilla/5.0 (compatible; Yahoo! Slurp;
> http://help.yahoo.com/help/us/ysearch/slurp)
>
> Mozilla/5.0 (compatible; Googlebot/2.1;
> +http://www.google.com/bot.html)


Desmond,
Ken has already given you good practical advice to which I have nothing
to add. But I'm wondering what you mean by saying your Web site is
"under attack". Yahoo! Slurp and Googlebot try to be reasonably polite
when spidering a site.

--
Philip
http://NikitaTheSpider.com/
Whole-site HTML validation, link checking and more

[Back to original message]