Fastest way to import data from one database to other in mysql using Java

Question

DBMS - mysql

Language - Java

I have two databases

Database One - I have only SELECT access.

Database Two - Complete access.

Exact Requirement : -

I have to import many TABLES from Database One to Database Two.

Current Solution :-

1. I am SELECTING all rows from table and storing result in ResultSet.

    String sql = "SELECT * FROM " + tableName +" ;"; Result Set res = runSql(sql); 

2. Write each row from the ResultSet into a CSV file.

    OutputStreamWriter writer = null; writer = new OutputStreamWriter(new FileOutputStream(fileName), "UTF-8"); while (res.next() ){ for (int j = 1; j <= noOfColumns.size(); j++) { if(j != 1) writer.append(","); writer.append("\\""+res.getString(j)+"\\""); } writer.append("\\n"); } 

3. Use LOAD DATA INFILE to fill database Two from CSV file.

    String sql = "LOAD DATA LOCAL INFILE '"+fileName+"' INTO TABLE "+tableName+ " CHARACTER SET UTF8 "+" COLUMNS TERMINATED BY ',' ENCLOSED BY '\\"' LINES TERMINATED BY \\"\\n\\";"; return runSql(sql); 

4. Repeating this process for all tables.

My Question :-

Can I do better as some tables have more than millions of rows? I find out that most of time is used by step 2 ( ResultSet to CSV writing ).

Answer 1

The immediate problem is that you should be using a BufferedWriter, but more to the point you should be creating the file at the server with SELECT ... INTO OUTFILE , not by iterating over a ResultSet or loading from a client-side file.

In fact why Java? The fastest way is not to use Java at all, but the MySQL Workbench, HeidiSQL, etc., and just import an SQL dump.