|
|
Posted by Jochem Maas on 08/12/05 03:20
Jasper Bryant-Greene wrote:
> Brian Dunning wrote:
>
>> On Aug 11, 2005, at 3:44 PM, Evert | Collab wrote:
>>
>>> Use robots.txt
>>> 'evil' searchengines will spoof the user-agent string anyway
>>
>>
>>
>> Can you be more specific about what you mean by "use robots.txt"?
>>
>> I just want to cloak for Google, MSN, and Yahoo. I couldn't care less
>> about what any other search engine (evil or not) does or sees.
>>
>
> robots.txt will not do what you want it to.
>
> Just sniff for those robots' User-Agents (Google, MSN and Yahoo all
> publish their UA strings on their websites, AFAIK) and send different
> content if it's one of those.
they will hammer you for it eventually - AFAICT all major SEs send out their
spiders occasionally with faked user-agent strings - to catch out crap like
this.
oh and the guy that invented php is a really bigcheese down at yahoo...
and he reads this list :-) though I doubt he has the time or desire to chase
you personally.
I would recommend you don't go down this road. it's bad for your business in the
longer term and its bad for the web because your filling it with shite.
>
> Jasper
>
Navigation:
[Reply to this message]
|