Transfer data from MySQL table to a different table

Question

I have a database, let's simply call it 'db', on my computer, with a few tables that have multiple columns and data inside those tables. I have a software using this database to store configuration elements and some other stuff.

Now, I am releasing a new version of my software, with only slight modifications in the database, ie some columns may have been added to tables, or removed (but no column renamed).

I must keep all data, so I would like to transfer it to the new "version" of my database.

What I thought of :

• Rename 'db' into 'db_old'.
• Install the new database as 'db_new', with the default values in the new columns
• For each table, get a list of all the columns from 'db_old' that are present in 'db_new'
• Use a INSERT INTO ... SELECT to put that old stuff back into 'db_new'.
• drop the old db and use my new db.

Do you think it can work ? Do you have any easy solution ?

Also, I'm absolutely not an SQL expert... And I tried this (without looking if the column has been removed or not yet) :

SELECT 
    GROUP_CONCAT(COLUMN_NAME
        SEPARATOR ',')
INTO @colList FROM
    INFORMATION_SCHEMA.COLUMNS
WHERE
    TABLE_SCHEMA = 'db_old'
        AND TABLE_NAME = 'configuration';
INSERT 
INTO db_new.configuration (SELECT @colList)
SELECT @colList FROM
    db_old.configuration;

But it fails on replacing the second @colList by the effective list... Can you also help me on this issue ?

Thank you everyone and have a nice day !

Answer 1

You should first take a dump of your DB Database and create a .sql file. Depending upon on your DB Data, this file can even go in GBs. This SQL File will contain all your tables and all the data inside those tables. I will suggest you open and see the file. Then you should use this new created file and use it to import all the data into new DB. It will put all those tables, data into this new DB.

Here is how to do that. First create SQL file:

mysqldump -h [SeverIpAddress] -u [UserName] -p[password] YourDbname > db_backup.sql

Use -h [SeverIpAddress] in case of Remote severs. In case, it resdies in your own system, you don't need to use this.

Then You should create your new DB, lets say DB_new. once created, switch to it using use command.

use DB_new

Once done, now import your .SQl file that we have created before using source command.

source YourSQLFilePath

In your case, source db_backup.sql

Answer 2

OK. If anyone ever encounter the same problem, here is the solution.

First, admit you have a database called 'myDatabase', with a table called 'myTable' that you want to "upgrade", ie you want to modify the table structure by adding/removing columns but keep the data inside.

---

First step is to drop foreign keys (if any) and to rename "myTable" :

USE `myDatabase`;

ALTER TABLE `myTable` DROP FOREIGN KEY `my_fk_constraint`;
ALTER TABLE `myTable` RENAME TO `old_myTable`;

---

Second step is to import the new table structure, by using SOURCE for example.

SOURCE C:/new_table_structure.sql

---

Third step is optional, but you may need this if your table has a lot of columns :

USE `myDatabase`;    
SET GLOBAL group_concat_max_len = 4294967295;

---

Fourth step is to store the following routine :

delimiter //

DROP PROCEDURE IF EXISTS updateConf//

CREATE PROCEDURE updateConf(IN dbName TEXT, IN old_table TEXT, IN new_table TEXT, IN primary_key_name TEXT)
BEGIN

-- get column count in old table
SELECT count(*)  
INTO @colNb 
FROM information_schema.COLUMNS 
WHERE TABLE_SCHEMA = dbName
AND TABLE_NAME = old_table;

-- get string with all column names from old_table
SELECT GROUP_CONCAT(COLUMN_NAME)
INTO @colNames1
FROM INFORMATION_SCHEMA.COLUMNS
 WHERE TABLE_SCHEMA = dbName
AND TABLE_NAME = old_table;
SET @colNames1 = CONCAT(@colNames1, ',');

-- get string with all column names from new_table
SELECT GROUP_CONCAT(COLUMN_NAME)
INTO @colNames2
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = dbName
AND TABLE_NAME = new_table;

    -- variables initialization
    SET @cpt = 1; -- column number counter
    SET @pos = 1; -- position of column name first char
    SET @vir = 1; -- next comma position
    -- start of loop
    label: LOOP
        IF @cpt <= @colNb THEN
            SET @vir = LOCATE(',',@colNames1,@pos); -- localize next comma
            SET @colName = SUBSTRING(@colNames1, @pos, @vir - @pos); -- get column name
            SET @pos = @vir + 1; -- update next column position
            -- if column is in both tables
            IF FIND_IN_SET(@colName, @colNames2) AND @colName != primary_key_name THEN 
                SET @execut = CONCAT("INSERT INTO ", new_table, " (", primary_key_name, ",", @colName, ") SELECT ", primary_key_name, ",", @colName, " FROM ", old_table, " ON DUPLICATE KEY UPDATE ", new_table, ".", @colName, " = ", old_table, ".", @colName);
                PREPARE stmt FROM @execut;
                EXECUTE stmt;           
            END IF;
            SET @cpt = @cpt + 1; -- counter increment
        -- when all columns parsed
        ELSE 
            LEAVE label; -- end of loop
        END IF;
    END LOOP label;
END //

delimiter ;

---

Final step is to call the procedure on tables, and to drop the temporary table:

CALL updateConf( 'myDatabase', 'old_myTable', 'myTable', 'primaryKeyName' );

DROP TABLE `old_myTable`;

And voila ! Just don't forget to put back the foreign keys you dropped :)

---

It surely can be done in better ways, but i got this to work correctly.

Thank you everyone !