|
|
Posted by Robert Cummings on 10/23/05 01:04
On Sat, 2005-10-22 at 17:36, Richard Lynch wrote:
> I've written a script to munge and import 108,000+ records.
>
> To avoid spiking the server, I'm sleep()ing 1 second for each record.
>
> So it takes 30+ hours to run, so what?
>
> This data changes "daily" but not really much more often than that,
> mostly.
>
> Anyway, it seems to be using an awful lot of RAM for what it's doing.
>
> Like, 80 Meg or so.
>
> php.ini memory_limit is set to 100M by my ISP. I can use my own
> php.ini and change that, if needed.
>
> Most of the fields are short, and the longest is maybe a varchar(255)
> and there are only ~15 fields.
>
> There's only a couple $query strings in each iteration, a couple MySQL
> result handles, and 2 copies (raw and munged) of each field's data.
>
> That don't sound like 80 Meg worth of data to this naive user.
>
> I got worried about the RAM, so stopped the process and added some RAM
> usage calls, started it over, and am logging the RAM usage at each
> record.
>
> I've written a "pretty" PNG graph and I'd like some experts to look at
> the graph, look at the code, and then tell me.
>
> 1. Do I have a memory leak that is gonna kill me and I have to fix it?
>
> 2. Is PHP's garbage-collection so non-aggressive that this is just
> "normal"?
>
> 3. Is there some kind of 80/20 or 90/10 "rule" in the guts of PHP
> garbage-collection, so that reducing my memory_limit would just "fix"
> this?
>
> 4. Can you spot any obvious/easy ways to alter the source to reduce
> memory usage without micro-managing or adding needless complications
> nor, perhaps most important, adding too much time onto the 30-hour
> process.
>
> Below is a link to the graph, some commentary, and there's a link to
> the PHP source code at the bottom-right of the web page.
>
> Hope all this isn't too presumptious...
The code looks fine, nothing being accumulated in the script itself so
my guess is that mysql is hanging onto something... possibly query
caching. Either that or as you say PHP itself though I can't say I've
experienced any huge memory issues in the past with long running
scripts. I'd definitely guess caching of some sort, especially since the
memory consumption levels out :)
Cheers,
Rob.
--
..------------------------------------------------------------.
| InterJinn Application Framework - http://www.interjinn.com |
:------------------------------------------------------------:
| An application and templating framework for PHP. Boasting |
| a powerful, scalable system for accessing system services |
| such as forms, properties, sessions, and caches. InterJinn |
| also provides an extremely flexible architecture for |
| creating re-usable components quickly and easily. |
`------------------------------------------------------------'
Navigation:
[Reply to this message]
|