|
|
Posted by "Richard Lynch" on 10/23/05 00:36
I've written a script to munge and import 108,000+ records.
To avoid spiking the server, I'm sleep()ing 1 second for each record.
So it takes 30+ hours to run, so what?
This data changes "daily" but not really much more often than that,
mostly.
Anyway, it seems to be using an awful lot of RAM for what it's doing.
Like, 80 Meg or so.
php.ini memory_limit is set to 100M by my ISP. I can use my own
php.ini and change that, if needed.
Most of the fields are short, and the longest is maybe a varchar(255)
and there are only ~15 fields.
There's only a couple $query strings in each iteration, a couple MySQL
result handles, and 2 copies (raw and munged) of each field's data.
That don't sound like 80 Meg worth of data to this naive user.
I got worried about the RAM, so stopped the process and added some RAM
usage calls, started it over, and am logging the RAM usage at each
record.
I've written a "pretty" PNG graph and I'd like some experts to look at
the graph, look at the code, and then tell me.
1. Do I have a memory leak that is gonna kill me and I have to fix it?
2. Is PHP's garbage-collection so non-aggressive that this is just
"normal"?
3. Is there some kind of 80/20 or 90/10 "rule" in the guts of PHP
garbage-collection, so that reducing my memory_limit would just "fix"
this?
4. Can you spot any obvious/easy ways to alter the source to reduce
memory usage without micro-managing or adding needless complications
nor, perhaps most important, adding too much time onto the 30-hour
process.
Below is a link to the graph, some commentary, and there's a link to
the PHP source code at the bottom-right of the web page.
Hope all this isn't too presumptious...
TIA!
http://l-i-e.com/feedbaby/memory_leak.htm
--
Like Music?
http://l-i-e.com/artists.htm
Navigation:
[Reply to this message]
|