Posted by Leonard Blaisdell on 07/25/06 01:00

In article
<NikitaTheSpider-2A1CFE.19502124072006@news-rdr-02-ge0-1.southeast.rr.co
m>,
Nikita the Spider <NikitaTheSpider@gmail.com> wrote:

> What makes you say that only a few robots support it? I had always
> assumed the opposite; that most robots support it. (Most decent ones,
> anyway -- the same that would respect robots.txt.)

I don't think robots are that difficult to create. I seem to remember
that I saw how to create a rudimentary one in a Perl book. If I wanted
to mine information from the net and was unscrupulous, I certainly
wouldn't worry about robots.txt and configure the robot to look for what
I wanted.
I think there are a pile of robots you don't see looking at your site if
it's available through httpd.conf or .htaccess holes. But then again,
I'm often wrong.

leo

--
<http://web0.greatbasin.net/~leo/>

[Back to original message]