Posted by Cleverbum on 12/13/06 14:06

Norman Peelman wrote:
> <Cleverbum@hotmail.com> wrote in message
> news:1165846992.423101.215040@f1g2000cwa.googlegroups.com...
> > I currently have a list of md5 strings and need to check if a new
> > string is in that list hundreds of thousands of times. I've found that
> > the fastest way to do this is to have all the md5's stored in an array
> > and use the php function in_array().
> > my only problem now is that populating this array with data from my sql
> > server is rather slow, I currently use the lines:
> >
> > $resone = mysql_query("SELECT * FROM logs_full");
> > mysql_close();
> >
> > while ($row = mysql_fetch_array($resone)) {
> > $md5array[$md5count]= $row['textmd5'];
> > $md5count++;
> > }
> >
> > to do this. does anyone have a faster method?
> >
>
> I can only shorten it:
>
> $resone = mysql_query("SELECT * FROM logs_full");
> mysql_close();
>
> while ($row = mysql_fetch_assoc($resone)) {
> $md5array[]= $row['textmd5'];
> }
>
> How big (how many rows) is the table you are reading in? Are you storing new
> md5's when they aren't found in the db?
>
>
> Norm
> --
> FREE Avatar hosting at www.easyavatar.com

The table currently has about 390,000 rows, and no I don't send the new
MD5s to it as soon as they are found to be new, I just put them into
the array.
I've found that moving away from single insert queries speeds up the
script many times over and so now I insert the data into the SQL tables
when I have 1000 new unique values.

[Back to original message]