Posted by Jochem Maas on 08/12/05 03:59

Philip Hallstrom wrote:
>>> robots.txt will not do what you want it to.
>>>
>>> Just sniff for those robots' User-Agents (Google, MSN and Yahoo all
>>> publish their UA strings on their websites, AFAIK) and send different
>>> content if it's one of those.
>>
>>
>> they will hammer you for it eventually - AFAICT all major SEs send out
>> their
>> spiders occasionally with faked user-agent strings - to catch out crap
>> like
>> this.
>
>
> google adsense won't. I explicity asked them about this. Well, what I
> asked was that if I had a password protected area, could I allow them
> access to spider the content so that normal users could see the ads. I
> told them the layout would be different, but the content the same.
>
> They said that was fine.

but you didn't ask - 'heh is it okay to fill my public page with SEO crud but
only if a spider comes round'

they might just take a different view on that :-)

>
> 2cents.
>

[Back to original message]