|
|
Posted by Norman Peelman on 12/13/06 16:35
<Cleverbum@hotmail.com> wrote in message
news:1166018776.381774.325990@f1g2000cwa.googlegroups.com...
>
> Norman Peelman wrote:
> > <Cleverbum@hotmail.com> wrote in message
> > news:1165846992.423101.215040@f1g2000cwa.googlegroups.com...
> > > I currently have a list of md5 strings and need to check if a new
> > > string is in that list hundreds of thousands of times. I've found that
> > > the fastest way to do this is to have all the md5's stored in an array
> > > and use the php function in_array().
> > > my only problem now is that populating this array with data from my
sql
> > > server is rather slow, I currently use the lines:
> > >
> > > $resone = mysql_query("SELECT * FROM logs_full");
> > > mysql_close();
> > >
> > > while ($row = mysql_fetch_array($resone)) {
> > > $md5array[$md5count]= $row['textmd5'];
> > > $md5count++;
> > > }
> > >
> > > to do this. does anyone have a faster method?
> > >
> >
> > I can only shorten it:
> >
> > $resone = mysql_query("SELECT * FROM logs_full");
> > mysql_close();
> >
> > while ($row = mysql_fetch_assoc($resone)) {
> > $md5array[]= $row['textmd5'];
> > }
> >
> > How big (how many rows) is the table you are reading in? Are you storing
new
> > md5's when they aren't found in the db?
> >
> >
> > Norm
> > --
> > FREE Avatar hosting at www.easyavatar.com
>
> The table currently has about 390,000 rows, and no I don't send the new
> MD5s to it as soon as they are found to be new, I just put them into
> the array.
> I've found that moving away from single insert queries speeds up the
> script many times over and so now I insert the data into the SQL tables
> when I have 1000 new unique values.
>
Ok, so then the next set of questions would be:
1) How large do you expect this table to get and how soon? (already consumes
12.5megs just for the textmd5 field alone)
2) How are you INSERTING the new info at the 1000 mark?
....what about LOAD DATA INFILE with the IGNORE switch?
http://dev.mysql.com/doc/refman/4.1/en/load-data.html
....or using array_diff()?
http://us2.php.net/manual/en/function.array-diff.php
Norm
Navigation:
[Reply to this message]
|