Posted by Newsgroups on 10/07/07 09:45

Thanks for your answers Nikita the Spider,


> If your aggregator is only being used by you and a few friends,
Currently, yes ;-( but I developped it also for anybody who want to use
it. :-)

> But if you want this aggregator to be used by lots of
> people, then I'd say you need to respect robots.txt.
The problem is : where is the limit between "few friends" and "lots of
people"...


> When describing robots, it focuses on indexing bots. But it was written
> at a time when Web robots were less varied then they are now, so the
> author may not have considered your case.
Yes, I agree. It's another debate, and I'm not used to reed rfc, so what
mean "Expires June 4, 1997" on this rfc ? Mean that Comments are not
considered after this date ? If not, I could comment this rfc. :-)

[Back to original message]