Posted by Sherm Pendley on 09/12/07 00:30

dorayme <doraymeRidThis@optusnet.com.au> writes:

> What is the simplest and most effective way of stopping robots
> searching a particular html pages on a server.

There are two popular "standards" (neither of which is a standard in
the formal sense). One uses <meta ...> elements in your HTML, and the
other uses separate robots.txt files. Both are described here:

<http://www.robotstxt.org/>

Both approaches depend on cooperative robots. For uncooperative robots,
all you can do is shout "klaatu barada nikto" and hope for the best.

sherm--

--
Web Hosting by West Virginians, for West Virginians: http://wv-www.net
Cocoa programming in Perl: http://camelbones.sourceforge.net

• Next in forum: Re: Stopping robots searching particular page
• Prev in forum: Re: Stopping robots searching particular page
• Thread view: Re: Stopping robots searching particular page

[Reply to this message]