|
Posted by CAH on 04/04/06 00:20
> IIRC, google and other sites search for a file called robots.txt that give
> directives on what it can and cannot index. Do a google search for
> robots.txt to see... (to verify, look in your webserver log files - it
> does show up as a request in my apache log files...)
>
> If your robots.txt includes the following directive - it will skip the
> entire site.
>
> User-agent: *
> Disallow: *
>
> or to limit the scope of it's search:
> User-agent: *
> Disallow: /cgi-bin/
> Disallow: /images/
> Disallow: *.php
I was testing this robot.txt
User-agent: Googlebot
Disallow: /*PHPSESSID
And that might solve it, I just do not know whether is works or not.
Mads
Navigation:
[Reply to this message]
|