Posted by Nikita the Spider on 10/08/07 01:02

In article <1191750311.5505.12.camel@localhost>,
Newsgroups <newsgroups@free.fr.invalid> wrote:

> Thanks for your answers Nikita the Spider,
>
>
> > If your aggregator is only being used by you and a few friends,
> Currently, yes ;-( but I developped it also for anybody who want to use
> it. :-)
>
> > But if you want this aggregator to be used by lots of
> > people, then I'd say you need to respect robots.txt.
> The problem is : where is the limit between "few friends" and "lots of
> people"...

That's where it gets tricky. =) But consider this -- if you obey
robots.txt 100% from the start, you'll always be doing the right thing
no matter how many people use your aggregator.

> > When describing robots, it focuses on indexing bots. But it was written
> > at a time when Web robots were less varied then they are now, so the
> > author may not have considered your case.
> Yes, I agree. It's another debate, and I'm not used to reed rfc, so what
> mean "Expires June 4, 1997" on this rfc ? Mean that Comments are not
> considered after this date ? If not, I could comment this rfc. :-)

That RFC was only a draft and it expired before it was approved.
However, no other RFC governing the use of robots.txt has ever been
approved or even written as far as I know, so that RFC is the closest
thing we have to a official standard.

--
Philip
http://NikitaTheSpider.com/
Whole-site HTML validation, link checking and more

[Back to original message]