Posted by Philip Hallstrom on 01/11/06 22:49

>>> I'm reading in a 66MB file into php so I can parse it and load it into a
>>> database. However, it is taking a long time for php to read the file.
>>> So long that the script times out. I know about the
>>> ini_set("max_execution_time", 120) or whatever length I want to set the
>>> time out for the script, but my question is as follows.
>>>
>>> Is it possible to read in a file and at the same time echo out a status
>>> of how far along the file has been read?
>>>
>>> The code I have right now is below. Just wondering if what I'm trying
>>> to do is possible and how to do it.
>>>
>>> // set script timeout
>>> ini_set("max_execution_time", 120);
>>>
>>> // import file name
>>> $log_file_name = "access_log.1";
>>> echo "Reading access log!<br/>";
>>> if (!$ac_arr = file("./$log_file_name")) {
>>> echo "Couldn't load access log!";
>>> die;
>>> }
>>>
>>> Thanks!
>>
>> Even if you get around the execution time the above code is going to use
>> up 66megs of memory to store that file in $ac_arr.
>>
>> You'd be much better off doing something like:
>>
>> $fd = fopen("./$log_file_name", "r");
>>
>> while ( !feof($fd) ) {
>> $buf = fread($fd, 32768);
>> // process buffer here
>> }
>> fclose($fd);
>>
>> That will let you do your progress thing as well as not eat up so much
>> memory. Play around with values of 32768 to see if smaller or larger
>> makes any different on your system.
>>
>> Good luck!
>
> Ahhh.. Okay that's very cool and I did not know that. I knew that it
> was chewing up a lot of memory reading it all at once.
>
> I guess the only thing I have to worry about now is if I do a read like
> that will it truncate a whole line instead of getting the whole line?
> The reason why I ask is that if possible I would rather have it read
> line by line for X lines and then process that buffer then go and grab
> more lines until end of file. Is that possible?

Sure. If it's a text file you could do something like this:

$i = 0;
$buf = "";
while ( !feof($fd) ) {
$buf .= fgets($fd);
if ( $i++ % 10 == 0 ) {
// process buf here
$buf = "";
}
}

Or you could turn buf into an array with $buf[] = fgets($fd) if that's
easier...

All kinds of options :)

-philip

[Back to original message]