|
Posted by NC on 10/09/46 11:38
Eric Anderson wrote:
>
> I have some files that sit on a FTP server. These files contain data
> stored in a tab-separated format. I need to download these files and
> insert/update them in a MySQL database. My current basic strategy
> is to do the following:
>
> 1) Login to the ftp server using the FTP library in PHP
> 2) Create a variable that acts like a file handle using Stream_Var in PEAR.
> 3) Use ftp_fget() to read a remote file into this variable (this is so I
> don't have to write it to disk).
> 4) Parse that data now stored in memory using fgetcsv() (again treating
> that variable as a file handle using Stream_Var). This produces an array.
> 4) Insert/Update the data in the array using DB in PEAR.
>
> This all seems to work and it means I don't have to write anything to
> disk.
This is not necessarily a good thing. Because you want minimize the
disk usage, you are missing out on MySQL's ability to process large
files very quickly. I would suggest an alternative approach:
1. Copy the remote file to your local disk.
2. Use LOAD DATA INFILE to load the data into MySQL.
3. Delete the data file, if necessary.
Cheers,
NC
Navigation:
[Reply to this message]
|