Dealing with bad CVS data and load data infile
Date: 06/27/07
(MySQL Communtiy) Keywords: no keywords
I use the following query to rip the contents of a CSV file into a temp table:
protected $loadfiledef = "
load data concurrent local infile '%s'
into table `%s`
FIELDS
TERMINATED by \",\"
OPTIONALLY ENCLOSED BY '\"'
LINES
TERMINATED by \",\r\n\"
IGNORE 1 LINES
;";
It's been pretty effective over running a script to parse individual lines, except for weird events where a field ends with a \ like so.
"JASMINE ESTATES\",80602,3,3,"RES"
NOTE: Above is only 4 fields out of 180 fields for that row.
That entire line ends up being mangled and causing corruption issues for all the following fields until the line is terminated.
I suppose I could try using sed or something similar to look for \", and replace it with \\", but I think this is just a bad value and something not checked by the IDX source.
Source: http://community.livejournal.com/mysql/115490.html