Posted by Nikita the Spider on 07/25/06 03:13

In article <1153790576.673362.138840@s13g2000cwa.googlegroups.com>,
"CRON" <cronoklee@hotmail.com> wrote:

> > What makes you say that only a few robots support it? I had always
> > assumed the opposite; that most robots support it. (Most decent ones,
> > anyway -- the same that would respect robots.txt.)
>
> I saw it on http://www.robotstxt.org/wc/exclusion.html but i think its
> mentioned in a few places. try a search for robots meta tag nofollow.

That page and all of the pages on robotstxt.org are very old. It is
still the closest thing there is to an authoritative standard, but only
because the standard hasn't changed much, not because the site's been
kept up to date.

Given that the majors (Google, Yahoo & friends) even support the
non-standard nofollow on individual links
(http://blog.searchenginewatch.com/blog/050118-204728), I think it is
safe to assume that they respect it when applied to the whole page.

Cheers

--
Philip
http://NikitaTheSpider.com/
Whole-site HTML validation, link checking and more

[Back to original message]