Inserting a MySQL result-set, retrieved from a select query?

Question

I've got a MySQL database located on server x and another on server y .

I'm trying to get the records from a table called testx which is located on server x, and INSERT them into a table called testy . So what I did was executed a SELECT statement and stored it into a resultset . Then I'm trying to iterate the INSERT statement within the resultset while loop. This is my sample code:

private static void cloneTableAndAlter() throws ClassNotFoundException, SQLException, InstantiationException, IllegalAccessException, InterruptedException {
        Connection connForSource = getConnectionForSource();
        Connection connForTarget = getConnectionForTarget();

        if (connForSource != null && connForTarget != null) {
            try {
                Statement st = connForSource.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
                        java.sql.ResultSet.CONCUR_READ_ONLY);

                String selectStatement = "SELECT field1, field2 FROM dbx.testx where time between ('2016-09-01 00:00:00') and ('2016-09-03 23:59:59');";

                String insertStatement = "INSERT INTO testy(" + "field1," + "field2)" + 
                        "VALUES(?,?) " +
                        "ON DUPLICATE KEY UPDATE field1 = VALUES(field1);"; <----field1 is a unique key but not primary

                st.setFetchSize(Integer.MIN_VALUE);

                ResultSet resultSetForSelect = st.executeQuery(selectStatement);
                PreparedStatement preparedStatement = connForTarget.prepareStatement(insertStatement);
                int count = 0;
                while (resultSetForSelect.next()) {
                    ++count;
                    TableDetails tableDetails = setTableDetails(resultSetForSelect); <--- i'm getting the value from the resultset and setting it to my DTO class.
                    getTableDetails(preparedStatement, tableDetails); <--- setting the values back during the insert from the DTO
                    preparedStatement.executeUpdate();
                    System.out.println(count + "rows affected");
                }
            } catch (SQLException e) {
                e.printStackTrace();
            }
        }
    }

NOTE : The source table( testx ) has more than a million records.

I'm able to do the insertion, but then I feel like the insertion is kinda slow where I'm getting 45-50 insertions per second. Where am I going laggy?

Is there any way that I could optimize this operation and increase the inserts or is this the nature of inserting a large data set?

Any help could be appreciated.

Answer 1

you can do on a other way:

Create a Select with this Concats

select GROUP_CONCAT(
    CONCAT(" ('",field1,"','",field2,"')")) as vals
from my_table;

The Result look like this:

Result

mysql> select GROUP_CONCAT(
    -> CONCAT(" ('",field1,"','",field2,"')")) as vals
    -> from my_table;
+---------------------------------------------------------------------------------------------------------+
| vals                                                                                                    |
+---------------------------------------------------------------------------------------------------------+
|  ('O1','AC'), ('O1','PT'), ('O2','PT'), ('O3','MI'), ('O3','PT'), ('O4','EG'), ('O4','PT'), ('O5','PT') |
+---------------------------------------------------------------------------------------------------------+
1 row in set (0,00 sec)

mysql>

and the you can direct concat this in the insert statement and you only read one row and only write and execute one statement.

your Code

    String selectStatement = "SELECT GROUP_CONCAT(CONCAT(" ('",field1,"','",field2,"')")) as vals
         FROM dbx.testx where time between ('2016-09-01 00:00:00') and ('2016-09-03 23:59:59');";

    ResultSet resultSetForSelect = st.executeQuery(selectStatement);


    String insertStatement = "INSERT INTO testy(" + "field1," + "field2)" + 
                            + String from result +
                            "ON DUPLICATE KEY UPDATE field1 = VALUES(field1);"; <----field1 is a unique key but not primary


# execute one time