Posted by StevePBurgess on 11/08/02 11:51

Hi thanks - but I don't want to prevent the search engines from
crawling my site - I just want a (relatively) accurate figure on how
many clicks through to the book site I've had.

I've written a script to check the user agent to see if it contains
bot, crawling,crawler,slurp etc as indicated by a previous poster. This
seems to work.


Thanks, all.

Steve


ctclibby wrote:

> Steve
>
> You should be able to take care of this in your robots.txt file ( root
> of your web directory ). I had a problem with a login link and the
> bots were getting to it and I would see one reference to the login
> file, but no POST. scratched me head for a bit... Anyway, Google it
> and you will find a wealth of info.
>
> todh

[Back to original message]