Posted by Philip Hallstrom on 08/12/05 03:52

>> robots.txt will not do what you want it to.
>>
>> Just sniff for those robots' User-Agents (Google, MSN and Yahoo all publish
>> their UA strings on their websites, AFAIK) and send different content if
>> it's one of those.
>
> they will hammer you for it eventually - AFAICT all major SEs send out their
> spiders occasionally with faked user-agent strings - to catch out crap like
> this.

google adsense won't. I explicity asked them about this. Well, what I
asked was that if I had a password protected area, could I allow them
access to spider the content so that normal users could see the ads. I
told them the layout would be different, but the content the same.

They said that was fine.

2cents.

[Back to original message]