How to copy data from another table without lock the table in MYSQL 6.2?

Question

I have a table that has all history data in Mysql server and it is very huge (about 700 million rows). I am creating a new table with the same columns but with partitioning, then I need to copy all data from the old table into the new partitioned table. I have already got the right script to do that but I think it might lock the table. I don't want that happen because it is on production server. What should I do to avoid locking the table?

Answer 1

Give that the tables have the exact same columns you can do something like this:

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED ;
INSERT INTO NEW_TABLE (SELECT * FROM OLD_TABLE);
COMMIT ;

I've included some additional explanation based on Wistar's comment. The read levels that can be used here are:

• READ COMMITTED : A somewhat Oracle-like isolation level with respect to consistent (nonlocking) reads: Each consistent read, even within the same transaction, sets and reads its own fresh snapshot
• READ UNCOMMITTED : SELECT statements are performed in a nonlocking fashion, but a possible earlier version of a row might be used. Thus, using this isolation level, such reads are not consistent. This is also called a dirty read. Otherwise, this isolation level works like READ COMMITTED.
• REPEATABLE READ : This is the default isolation level for InnoDB. For consistent reads, there is an important difference from the READ COMMITTED isolation level: All consistent reads within the same transaction read the snapshot established by the first read. This convention means that if you issue several plain (nonlocking) SELECT statements within the same transaction, these SELECT statements are consistent also with respect to each other.
• SERIALIZABLE : This level is like REPEATABLE READ, but InnoDB implicitly converts all plain SELECT statements to SELECT ... LOCK IN SHARE MODE if autocommit is disabled. If autocommit is enabled, the SELECT is its own transaction. It therefore is known to be read only and can be serialized if performed as a consistent (nonlocking) read and need not block for other transactions. (To force a plain SELECT to block if other transactions have modified the selected rows, disable autocommit.)

I hope this helps.

Answer 2

I do not know what is your script but I suggest you insert with chucks. See this example .

If you use SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED you might incorrect version of your row being inserted. If you insert all the rows with one selects, well you will not only have locks but it won't be the best performance wise.

You could do something like

 INSERT INTO NEW TABLE (SELECT * FROM OLD_TABLE LIMIT 1000 OFFSET 0 )
 INSERT INTO NEW TABLE (SELECT * FROM OLD_TABLE LIMIT 1000 OFFSET 1000 )
 INSERT INTO NEW TABLE (SELECT * FROM OLD_TABLE LIMIT 1000 OFFSET 2000 )
 ...

or use prepare statement and while

CREATE PROCEDURE myproc()
BEGIN
    @rows :=0
    SELECT COUNT(*) FROM OLD_TABLE into @rows
    DECLARE i int DEFAULT 0;
    WHILE i <= @rows DO
        PREPARE stmt1 FROM 'INSERT INTO NEW TABLE (SELECT * FROM OLD_TABLE LIMIT 1000 OFFSET ? )'
        EXECUTE stmt1 USING @i;
        DEALLOCATE PREPARE stmt1;
        SET i = i + 1000;
    END WHILE;
END

Of course, you can adjust the chunk size according to your config by changing the LIMIT size

Answer 3

Copy in chunks. Do you have an AUTO_INCREMENT PRIMARY KEY ? If so then do

 WHERE id >= $x AND id < $x + 1000

If there are lots of gaps, or if you have other issues, then see other techniques for efficiently chunking .

The evils of Pagination via OFFSET .

Better yet, is to use pt-online-schema-alter from Percona. It does most of the thinking behind what I described, plus it allows you to write to the table while the copy is being done. (It uses TRIGGERs to achieve it.)

Answer 4

To reduce the disadvantages of using OFFSET , this article describes a possible approach to use a JOIN when a numeric primary key id is available to force the use of the proper index. Note that to keep track of process, a "procedure_log" table is created and gradually updated after processing a batch:

For MySQL:

DROP PROCEDURE IF EXISTS copyTable;

DELIMITER |
CREATE PROCEDURE copyTable()
BEGIN

    DECLARE batchSize INT DEFAULT 100;
    DECLARE i INT DEFAULT 0;
    DECLARE rowCount INT;

    # Note that we use a WHERE clause to prevent a full table scan / use the index properly
    SET rowCount = (SELECT COUNT(id) FROM my_table WHERE id IS NOT NULL);

    CREATE TABLE IF NOT EXISTS my_table_copy LIKE my_table;
    CREATE TABLE IF NOT EXISTS procedure_log ( id INT UNSIGNED NOT NULL AUTO_INCREMENT, entry TEXT, timestamp DATETIME DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (id) );

    WHILE i <= rowCount DO
        INSERT IGNORE INTO my_table_copy (
          SELECT source.* FROM (
            SELECT id FROM my_table ORDER BY id LIMIT i, batchSize
          ) tmp
          JOIN my_table source ON source.id = tmp.id
          ORDER BY source.id
        );
        SET i = i + batchSize;

        INSERT INTO procedure_log (entry) VALUES (CONCAT('Copied batch from my_table => my_table_copy, batch: ', batchSize, ', offset: ', i, ', rowCount: ', rowCount));
    END WHILE;
END |
DELIMITER ;