|
|
Posted by Richard Levasseur on 06/11/06 22:38
I wouldn't be surprised at all if PHP was loading the whole file into
memory or scanning from start to finish every fgetcsv() call. Checking
the C source code would give better insight.
there are also a couple functions that will tell you the current memory
usage by PHP. Putting those in for each iteration might give some
insight into if its running out of memory.
Maybe it is the data confusion the function, too? Unescaped quote,
missing newline, too long of a line, etc. Something like that might
confuse the fgetcsv() function.
tony@tony.com wrote:
> In article <xPxig.3721$sC.2396@fe24.usenetserver.com>,
> david.haynes2@sympatico.ca says...
> >
> > A slow down of this magnitude typically points to some system-related
> > bottleneck rather than an algorithmic one. Have you checked the
> > processes' virtual memory use? I would suspect that you are starting to
> > swap around the 6,000th record.
> >
> > If not, I would start to place finer-grained time information around the
> > major I/O points (fgetcsv, fwrite) to see if they are causing the slow down.
> >
> > On a stylistic note, why do you use $x++ in some places and $x += 1 in
> > others? Also, the checkit function could use the trinary compare operator:
> >
> > function checkit($star) {
> > exec('declination.exe '.$star, $aout, $returnval);
> > return ($aout[0][0] === 'Y');
> > }
> >
> > and 'if( $checkit($star) == false ) {' could become
> > 'if( ! $checkit($star) ) {'
> >
> > However, I don't think any of these would contribute to your slow down
> > issue.
> >
> > -david-
> >
>
> Thanks for the comments david - I've run this on both windows and linux
> now and the linux system I ran through apache - I get the same results on
> that too (very similar but not identical) (The linux box is entirely
> different hardware) - I'll post timing differences later - they're
> outrageous !!!
>
> I've also run the windows version with apache too now and still get the
> slowdown.
>
> I've tried removing my call to the executable replacing it with a simple
> return and it still slows down.
>
> Looking at what you suggest - could it be that the fgetcsv command is
> searching from the top of the file on every itteration?
> I dont cause it to - but if thats how it works ?
>
> I have discovered a windows loss of 2Mb of system memory for every run -
> thats either a windows or PHP problem I dont know which - probably
> windows and I dont think its connected to the slowdown.
>
> If this isnt obvious to anyone I guess the only thingI can do is start
> taking things out one by one starting with your suggestions.
>
> On the memory side - no it all(suprisingly) seems to happen in ram - even
> the test database file of 1 million records seems to go straight to ram.
> There's no thrashing or anything (The win-98 system has 1Gb ram - the
> linux system just 128Mb)
>
> On the style front - Dave you wouldn't believe my working methods!
> In my time I've used maybe 7 languages in anger and currently I use 3 or
> 4 so I'm in and out of them all the time - things get mixed up in my
> head.
> I always do a "comment" sweep when I'm done and tidy things up for this
> very reason but sometimes I do use ++x and += 1 because it helps me
> remember whats going on and what I need to keep an eye on. (I have a bad
> short term memory problem) My "if" statements and other stuff follow the
> same routine for the same reasons - there is method in my madness - you
> just have to be mad too to see it...
> This particular code has been messed about with quite badly too as I try
> to find the problem.
>
> I care more about function not fancy - most PHP style I've seen is
> abysmal anyway - the standard style reccomended is appaling. I'm not a
> fan of the way most people code - nor of the way many languages work for
> that matter. Anything other than Forth is bad form in my book ;-)
>
> As for the trinary operator - its a terrible construct for anyone who
> doesn't use it regularly or anyone looking at someone elses code so I
> avoid it.
>
> tony
Navigation:
[Reply to this message]
|