How to convert about 16 million rows in mysql with least time?

Question

I'm having about 1.6 million rows in a table having only two columns one is bigint unique index and another is longblob having photo in base64. I need to decode these base64 photos as soon as possible. I tried to use the java program which interrupts after few hours and the speed is also not good. Here is the program

Connection conn = null;
        Statement stmt = null;
        try {
            conn = DatabaseConnection.getConnection();
            stmt = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE,
                    ResultSet.CONCUR_UPDATABLE);
            ResultSet uprs = stmt.executeQuery("SELECT uniqueid,photo FROM newphotodata");
            String query="insert into photo_data values (?,?)";
            PreparedStatement pstmt =  conn.prepareStatement(query);
            while (uprs.next()) {
                byte[] processed = Base64Utils.base64Decode(uprs.getString(2));
                pstmt.setString(1, uprs.getString(1));
                pstmt.setBytes(2, processed);
                pstmt.executeUpdate();
                uprs.deleteRow();
            }
            pstmt.close();
        } catch (SQLException e) {
            e.printStackTrace();
        } finally {
            try {
                if (stmt != null)
                    stmt.close();
                if(conn!=null)
                    conn.close();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

The second approach I've taken is using database trigger which uses base64_decode stored procedure I've created. Here is the trigger

DELIMITER $$

DROP TRIGGER `delete_photo`$$

CREATE TRIGGER `delete_photo` BEFORE DELETE ON `newphotodata`
FOR EACH ROW
BEGIN
    INSERT INTO `photo_data` SELECT OLD.`uniqueid`, BASE64_DECODE(OLD.`photo`);
END$$

DELIMITER ;

The speed of processing is again too slow. Is there any other method which can be used to perform the following task. I'm running Mysql version 5.0 in Redhat Linux with 96GB RAM, intel Xeon x5672.

Answer 1

The best way is create another table and insert the decoded photo using the pre built mysql function to decode encode on base64.

Inserting is faster than updating.

INSERT INTO photo_data 
SELECT OLD.uniqueid, FROM_BASE64(OLD.`photo`);

But this query can become very long if you don't commit every few rows. So the best way is create a procedure that make a commit every n rows.

Don't exit from mysql to have the best performances. Each transformation made outside mysql needs additional efforts.

Edited: Order the original data by uniqueId. If a problem happens you can restart from the last inserted id. You don't need to delete the original data. It improves your performances.

Answer 2

For the java-program you can try using a sql batch statement, that will drastically improve the speed of the insert-statements. Example code:

int batchLimit = 1000;
int currentBatchLimit = batchLimit;
while (rs.next())
{
    stmt.setInt(1, 123);
    stmt.addBatch();

    currentBatchLimit--;
    if (currentBatchLimit == 0)
    {
        stmt.executeBatch();
        stmt.clearBatch();
        currentBatchLimit = batchLimit;
    }
    stmt.clearParameters();
}
stmt.executeBatch();
stmt.close();