Posted by John Cage on 02/17/05 10:59

Hi there

Just an update. We altered our code last night and the result is that
we have managed to bring in everything with no problem now. The slowest
part was actually pulling down the emails which are on another server.
I'd like to thank everyone for their help

I have another project I'm working on - more as a hobby than anything
else. I need to create a crawler that will crawl approximately 100,000
domains, pull in the information and then classify the content based on
some logic (still to be decided but thinking of using bayesian
filters). Is there anyone on this list who has written fast and decent
crawlers in PHP who would be willing to share their experiences?

Thanks again for your help - it was much appreciated

John

• Next in thread: Re: Crawlers (was parsing large files - PHP or Perl)
• Next in forum: md5 passwords to db
• Prev in forum: Problem connecting to MySQL with PEAR::DB and mysqli
• Thread view: Crawlers (was parsing large files - PHP or Perl)

[Reply to this message]