Posted by Matthew Weier O'Phinney on 08/12/05 04:00

* Brian Dunning <brian@briandunning.com> :
> On Aug 11, 2005, at 4:06 PM, Evert | Collab wrote:
>
> > First hit on google:
> > http://www.searchengineworld.com/robots/robots_tutorial.htm
> > Search engines check for a robots.txt on your site, in the
> > robots.txt file you can specify that certain or all search engines
> > shouldn't index your site
>
> I know what robots.txt is, I meant how would you use that to cloak
> the site. Put PHP code in robots.txt to log the IP of any requests
> to a db, and then use that db to cloak the rest of the site or not?

If you want to dynamically determine what to disallow based on the
UserAgent string, simply tell Apache, via an .htaccess file, to pass
robots.txt to PHP for handling. Then have that script do the processing
and return output compatible with the robots.txt specification.

--
Matthew Weier O'Phinney
Zend Certified Engineer
http://weierophinney.net/matthew/

[Back to original message]