Posted by tony on 06/10/06 10:06

I'm using PHP 5 on Win-98 command line (ie no web server involved)

I'm processing a large csv file and when I loop through it I can process
around 275 records per second.

However at around 6,000 records this suddenly drops off to around 40
records per second.

This is a big problem as the "live" list is over 4 million records long.
I'd break it up but this is to be a regular test so that would be messy
to say the least -

Each record is 8 fields & total length tends to be below 200 characters
CSV is comma and ""

I was wondering if anyone with strong PHP knowledge has heard of this or
could help explain it please (As you probably know I'm very new to PHP)

I've trimmed the startup code to pseudocode to make it easier to read.
Otherwise my code is as below:

Sorry if line wrap is wrong - that would be my newsreader not the code

As you can see the code grabs a field from the database - spawns a
windows (msdos command line) .exe file to test it and writes the field
out to either a good or bad result file.

I dont do any file seeking or open and closing of files during the loop.

Tony

------------------------------ CODE START ------------------

<?php

//+++++++++++++++++++++++++++ PSeudocode start
open all new files for appending here (fopen($fin, 'a');)
open database for read-only here
Initialise all variables to 0 here

START;
get start-time
loop()
get end-time
write-statistics
close all files here
exit;
// +++++++++++++++++++++++++PSeudocode End

function loop() {
global
$fin,$fout,$fgood,$records,$fields,$good,$bad,$total,$dif_fcount,$nodata
;
while (($data = fgetcsv($fin, 1024, ",", "\"")) !== FALSE) {
if($data == '') { continue; }
$records++;
if (count($data) != $fields ) { $fields = count($data);
$dif_fcount++; }
if ($data[2] == '') { $data[2] = 'NO DATA' ; $nodata++; }
$raw = $data[7];
$star = "\"" . ($data[2]) . "\"";
$star = $raw ;
if (checkit($star) == false) {
fwrite($fout, $records . "," . $raw . "\r");
$bad += 1;
} else {
fwrite($fgood,$star . "\r");
$good += 1;
}
$total += 1;
echo("Total checked: " . $total . "\r" );
} //while
}


function checkit($star) {
exec("declination.exe " . $star , $aout, $returnval);
if ($aout[0][0] === "Y") {
return true;
} else {
return false;
}
}
?>

[Back to original message]