|
|
Posted by Phil Earnhardt on 11/22/55 11:39
On Tue, 07 Feb 2006 20:28:58 -0500, Barry Margolin
<barmar@alum.mit.edu> wrote:
>> I can't imagine how you would categorically block them. OTOH, the
>> Robots Exclusion Protocol can be used to tell anyone who honors such
>> things that you don't want your website copied.
>
>I wouldn't expect a manual download application to honor it. That
>mechanism is intended to control automated web crawlers, like the ones
>that Google uses to index all of the web.
wget respects the Robot Exclusion Protocol; curl does not.
--phil
Navigation:
[Reply to this message]
|