|
|
Posted by "Richard Lynch" on 10/23/05 03:49
On Sat, October 22, 2005 5:04 pm, tg-php@gryffyndevelopment.com wrote:
> Does it make a difference if instead of one record at a time, you pull
> 10, 20, 100... might speed up your process a little and if it
> doesn't use any more memory, then why not?
I could make a buffer of N lines and read them, I guess.
I kinda figured at 1 line a second, PHP and the OS would have no
problem buffering from the disk.
> Another thought.. can you use database replication here? So instead
> of dumping all the records and processing and inserting the ones that
> are new, you'd only be acting on what's changed since the last backup.
> With real replication, you can do this realtime (or at least
> semi-realtime), but you could simulate replication in your PHP script
> as well.
This data's coming from an external source.
Replication is probably not gonna happen.
Though I am going to ask for a timestamp column so I can ignore rows
that haven't changed since my last import.
That should kill about 90% of the MySQL processing, which seems to be
the biggest CPU sink here.
If that happens, I can probably change the sleep to usleep and a
quarter second or so.
--
Like Music?
http://l-i-e.com/artists.htm
Navigation:
[Reply to this message]
|