Posted by Ken Sims on 08/23/06 14:49

Hi Paul -

On 23 Aug 2006 03:34:44 -0700, "Paul" <desotuatail@aol.com> wrote:

>The website is www.des-otoole.co.uk

You need a robots.txt text file at the root of the site (e.g.
accessible as <www.des-otoole.co.uk/robots.txt>).

See http://www.robotstxt.org/wc/norobots.html

This robots.txt file tells all robots to not access any part of your
website:

User-agent: *
Disallow: /

Of course bad robots won't bother to even retrieve the file or will
retrieve it and ignore it, but that's another issue.

Google, Yahoo, MSN, etc. will retrieve and obey the robots.txt (though
you may still see some activity for a little while since they use
multiple servers for indexing and it may take a while for any given
server to retrieve an up-to-date copy of robots.txt).

--
Ken
http://www.kensims.net/

[Back to original message]