Posted by Jonathan N. Little on 10/08/06 23:06

Nikita the Spider wrote:
> In article <2fc8a$4527add2$40cba78b$5768@NAXS.COM>,
> "Jonathan N. Little" <lws4art@centralva.net> wrote:
>
>> And to misbehaving spiders, if they don't play by the rules then there
>> is not much you can do...except maybe redirect in an .htaccess entry for
>> the little bugger. But you would have to trap the user agent of each
>> misbehaving spider.
>
> I'm sure you're aware that misbehaving spiders can also send a false
> user-agent string. There's nothing (except the threat of legal action)
> to stop them from masquerading as Googlebot, for instance. That said, a
> lot of ill-behaved spiders that I see in my server logs look like they
> send distinctive user-agent strings.
>

As I said "if they don't play by the rules then there is not much you
can do"

--
Take care,

Jonathan
-------------------
LITTLE WORKS STUDIO
http://www.LittleWorksStudio.com

[Back to original message]