|
Posted by Nick Theodorakis on 09/12/07 14:04
On Sep 12, 5:55 am, Ben C <spams...@spam.eggs> wrote:
>
> >
> 2. You forget to turn off the webserver's AutoIndex or similar, so the
> spider can just navigate its way to the url going through auto
> generated directory indexes.
At least one robot does this. I have a template page (definitely not
mentioned anywhere else) in a subdirectory that seems to get spidered
by the yahoo slurp robot.
Nick
--
Nick Theodorakis
nick_theodorakis@hotmail.com
Navigation:
[Reply to this message]
|