|
|
Posted by Barbara de Zoete on 02/07/06 21:44
[ like to block grabbers/downloaders and the like ]
On Tue, 07 Feb 2006 20:34:36 +0100, <cmashieldscapting@hotmail.com> wrote:
> Phil Earnhardt wrote:
>
>> I can't imagine how you would categorically block them. OTOH, the
>> Robots Exclusion Protocol can be used to tell anyone who honors such
>> things that you don't want your website copied.
>>
>
> Try doing a search using key topic words on Usenet, or at least Google
> Groups, the Google version of Usenet. I'm pretty sure I saw a couple
> of discussions regarding this.
>
I found this site <URL:http://www.psychedelix.com/agents/index.shtml> that
lists the 'handles' (don't know a better word for that right now) of know
bots and user agents et cetera. I singled out the ones marked with
D(ownload) and S(pam bot or other bad bot) and put them in my httpd.ini
file[1] to redirect them into nothing. This is all relatively new to me
though, so I'll have to see in my logs if the next Mget try succeeds, or
not :-)
[1] I'm on a IIS server
--
______PretLetters:
| weblog | http://www.pretletters.net/weblog/weblog.html |
| webontwerp | http://www.pretletters.net/html/webontwerp.html |
|zweefvliegen | http://www.pretletters.net/html/vliegen.html |
Navigation:
[Reply to this message]
|