|
|
Posted by Leif K-Brooks on 10/22/06 19:37
Nikita the Spider wrote:
> My guess is that this will probably work with most bots, but it isn't a
> sure thing. Oddly enough, robots.txt is not as clearly standardized as
> HTML or HTTP.
Nor should it be in its current state. It's very poorly-implemented. The
idea of using a resource for this purpose is ridiculous: asking for
permission to use a resource is a metadata issue; it should be handled
with a special HTTP header, or something similar.
Using a resource for other resources' metadata has practical problems;
for instance, Wiki software which lets users create their own files has
to special-case the name 'robots.txt'. If they don't do that, a
malicious user could, through the standard Wiki interface, cause search
engines to ignore the site. Robots.txt is broken.
Navigation:
[Reply to this message]
|