MySQL LOAD DATA INFILE with fields terminated by non-ASCII character

Question

I have a lowercase thorn separated file that I need to load into a MySQL database (5.1.54) using the LOAD DATA INFILE ... query.

The file I'm trying to load is located on the same server as the MySQL database, and I'm issuing the query from a Windows machine using SQLYog , which uses the MySQL C client library.

I'm having some major issues, I've tried using the FIELDS TERMINATED BY 0x00FE syntax using all the variations of the thorn character I can think of, and I've tried changing the character set of the connection ( SET NAMES ... ), but I consistently get the warning...

Warning Code : 1638
Non-ASCII separator arguments are not fully supported

...and all the data loads into the first column.

Is there any way around this at all? Or am I resigned to pre-processing the file with sed to replace all the thorn 's with a more sensible character before loading?

Answer 1

I have succeeded to load this data with Data Import tool (CSV format) in dbForge Studio for MySQL . I just set 'Þ' as custom delimiter. The import from the CSV format is fully supported in free Express Edition.

Answer 2

I decided to fix the file by replacing the non-ASCII character with a character that MySQL's LOAD DATA INFILE ... would understand.

1. Use od to get the octal byte value of the offending character - od -b file.log - in this case it's 376 .

2. Use grep to make sure the character you want to replace it with doesn't already exist in the file - grep -n '|' file.log grep -n '|' file.log .

3. Use sed and printf to replace the non-ASCII character - sed -i 's/'$(printf '\\376')'/|/g' file.log .