Posted by Nikita the Spider on 10/08/06 18:42

In article <2fc8a$4527add2$40cba78b$5768@NAXS.COM>,
"Jonathan N. Little" <lws4art@centralva.net> wrote:

> And to misbehaving spiders, if they don't play by the rules then there
> is not much you can do...except maybe redirect in an .htaccess entry for
> the little bugger. But you would have to trap the user agent of each
> misbehaving spider.

I'm sure you're aware that misbehaving spiders can also send a false
user-agent string. There's nothing (except the threat of legal action)
to stop them from masquerading as Googlebot, for instance. That said, a
lot of ill-behaved spiders that I see in my server logs look like they
send distinctive user-agent strings.

--
Philip
http://NikitaTheSpider.com/
Whole-site HTML validation, link checking and more

[Back to original message]