Posted by Philip Ronan on 06/29/05 15:05

"studio@stoxfx.com" wrote:

> I have a simple crawler as part of a website. It uses file() to copy
> HTML content from a web page on another website and reformat it into a
> new HTML page using a template. Is there a simple way of having the
> crawler supply a custom HTTP user agent header to identify itself to
> the website being crawled? I don't want to alter the php.ini file, I
> need to set the user agent value from PHP if possible.
>
> Thanks
>

There may be safe-mode restrictions to prevent you doing this, but have a go
anyway:

ini_set("user_agent","Marvin the Paranoid Android");

If you're planning to crawl sites that don't belong to you, be nice and
check the robots.txt file first.

--
phil [dot] ronan @ virgin [dot] net
http://vzone.virgin.net/phil.ronan/

• Next in forum: DB::connect and PHP : hostname seems to be ignored ?!?!
• Prev in forum: Re: custom user agent
• Thread view: Re: custom user agent

[Reply to this message]