Posted by Andy Dingley on 07/25/06 09:20

CRON wrote:

> Found this:
> <meta name="robots" content="noindex, nofollow">
> Apparantly only a few robots support it. Anyone know which ones?

It's not a question of support, it's a question of meaning. The <meta>
tag obviously only applies to that page, and links from that page. It's
also available before needing to retrieve any pages. robots.txt applies
to whole directories and is processed before retrieving pages.

They're complementary, not substitutes. Use both if you wish. Certainly
use robots.txt

[Back to original message]