Posted by StevePBurgess on 12/18/02 11:51

Hi thanks - but I don't want to prevent the search engines from
crawling my site - I just want a (relatively) accurate figure on how
many clicks through to the book site I've had.

I've written a script to check the user agent to see if it contains
bot, crawling,crawler,slurp etc as indicated by a previous poster. This
seems to work.


Thanks, all.

Steve


ctclibby wrote:

> Steve
>
> You should be able to take care of this in your robots.txt file ( root
> of your web directory ). I had a problem with a login link and the
> bots were getting to it and I would see one reference to the login
> file, but no POST. scratched me head for a bit... Anyway, Google it
> and you will find a wealth of info.
>
> todh

• Next in forum: Re: if/else statement, form-check
• Prev in forum: Re: PHP email function
• Thread view: Re: Checking if referrer is web crawler

[Reply to this message]