如何使用Java将特殊字符插入MySQL

Question

I have several csv files in am loading into MySQl using Java. In the Description field I have several Special Characters that are causing the load to fail. I am using LOAD DATA INFILE as seen in the code block below. This is nested in a for each loop which parses an array of filenames / tables and runs through each combination until it is finished with all the files.

Here is my jdbc connection string where I am passing a definitive collation param/value for UTF8 collation

 static String  url = "jdbc:mysql://localhost:3306/iber_stage?verifyServerCertificate=false&characterEncoding=UTF8";

other connection parameters and parsing an array of filenames/tablenames

 final String sql1 = ("TRUNCATE TABLE" + tableName);
 final String sql2 = ("LOAD DATA INFILE" + filetoEat  + "INTO TABLE staging." +tableName + "CHARACTER SET UTF8 FIELDS TERMINATED BY',' ENCLOSED BY '\"\' LINES TERMINATED BY '\n' IGNORE 1 LINES");

        try {
        Class.forName("com.mysql.jdbc.Driver");
        con = DriverManager.getConnection(url, username, password);
        st = con.createStatement();
        st.executeUpdate(sql1);
        rs = st.executeQuery(sql2);

        if (rs.toString() != null) {
            returnMsg = rs.toString();
            System.out.println(returnMsg);        
            updFlag = 0; 
            String strRecs = returnMsg.substring(40);
            updateControlTable(updFlag, strRecs);
        }

        } catch (SQLException ex) {
            Logger lgr = Logger.getLogger(update.class.getName());
            lgr.log(Level.SEVERE, ex.getMessage(), ex);
            updFlag = 1;            

        } catch (ClassNotFoundException e) {
            Logger lgr = Logger.getLogger(update.class.getName());
            lgr.log(Level.SEVERE, e.getMessage(), e);
            e.printStackTrace();
            updFlag = 1;

        } 

The code is working fine until it comes across a special character like a degree symbol or micro symbol µ within a Material Description . At that point it throws an Exception

Invalid utf8 character string: 'LUG'

The string LUG is followed by a µ symbol. The DB is set to utf8 - utf8_unicode_ci and the column in question is a VARCHAR(60) that holds material descriptions. I have tried using ESCAPED BY '\\\\' but I can't seem to get it working correctly. I have also tried CHARACTER SET UTF8 . I have also tried different collation ie, utf8_general_ci to no avail.

Any insight is greatly appreciated

Answer 1

Have you tried adding

CHARACTER SET UTF8

to the LOAD DATA INFILE instruction?

Full doc: http://dev.mysql.com/doc/refman/5.7/en/load-data.html

Answer 2

Can you check with database collation utf8_general_ci and character set as utf_8 , It may work for you.

As It applies Unicode normalization using language-specific rules.

Answer 3

I figured that I would answer this now that I found the solution. Because I am using Java to run the LOAD DATA INFILE via JDBC the JDBC driver seems to be checking the collation at the DB and not the actual table being loaded as it is parsing the file. So you can't have the DB set to UTF-8 and have a Latin collated table as you would be able to do with an INSERT statement. I had tried to set the Table collation as Latin and even had the field in question Latin, but until I changed the entire DB to Latin it was failing. The CSV files are large so checking every char in question is not easy, but I was catching the Exceptions in Java and was able to determine that the error was generated by the JDBC driver and was complaining that "Character at line xx is not a UTF-8 character" Running in Debug allowed me to see more details.

I then concluded it must not be looking at the Latin collated table it would be filling, but was looking at the DB which was still set to UTF-8. Changing the DB to Latin was all I needed to do.

I hope this will help others in the future.

Pat