How do I update a MySQL database with the data from another database?

Question

Disclaimer: this is a bit of a "best practices" question, but I'm not sure how else to phrase it to be more concrete or evoke a more objective answer.

I have two databases for a small Rails project: a dev database and a prod database that live on the same server.

What happens is that every couple weeks, I make some data changes (mostly inserts) via the ActiveAdmin gem to the dev database via a dev Rails environment. I poke around the Rails app making sure the new data looks good. When I'm ready to deploy, I run a script that:

1. Dumps the dev db
2. Drops and recreates the prod db to delete all data
3. Imports the dev db dump into the prod db

My intuition tells me that this is not a great way of going about this, and it's also a bit slow, but I can't seem to find the standard way of doing data deployments from one database to another. What is the standard way, if there is one?

Things I've considered:

• Setting up an additional database that is the replica of the dev database; in deployment, somehow switch the Rails app over to use the replica as "prod", update the old "prod" db to be the replica, etc. I can barely even keep this idea in my head, and it seems like such a mess.
• Doing data changes directly on prod, which completely invalidates the need for a dev database (feels very gross)
• Doing data changes in dev via SQL script with transactions, and applying them to prod when deploying (it'd be really, really annoying to write these scripts by hand)

Some additional notes:

• The only data changes that are ever made are the ones I make
• Schema changes are done via Rails migrations
• The databases are relatively small (the biggest table is ~1000 rows)

Answer 1

If you have two databases on the same server, you can compare and insert into tables. First, for deleted rows:

BEGIN TRAN;
DELETE FROM prod.tbl1
WHERE id IN (
 SELECT id FROM dev.tbl1 RIGHT JOIN prod.tbl1 ON dev.tbl1.id = prod.tbl1.id WHERE dev.tbl1.id IS NULL);
COMMIT;

Second, for new rows:

BEGIN TRAN;
INSERT INTO prod.tbl1
SELECT *
FROM dev.tbl1
WHERE id IN (
 SELECT id FROM dev.tbl1 LEFT JOIN prod.tbl1 ON dev.tbl1.id = prod.tbl1.id WHERE prod.tbl1.id IS NULL);
COMMIT;

Now, a trigger on your dev database to manage updates:

CREATE DEFINER=`root`@`localhost` TRIGGER `dev`.`tbl1_update`
AFTER UPDATE ON `dev`.`tbl1`
FOR EACH ROW
BEGIN
    SET NEW.update = '1';
END

You need a "update" field on the dev table. When a update query run on the table, the field "update" changes to 1 automatiaclly. Then, use this query:

BEGIN TRAN;
UPDATE prod.tbl1
LEFT JOIN dev.tbl1
    ON prod.tbl1.id = dev.tbl1.id
SET prod.tbl1.fld1 = dev.tbl1.fld1, prod.tbl1.fld2 = dev.tbl1.fld2
WHERE prod.tbl1.id IN (SELECT id FROM dev.tbl1 WHERE update = '1');
UPDATE dev.tbl1 SET update = '0';
COMMIT;

You can run a query like this on all tables. You can put it on a .sql file and run with a cron job (mysql -h -u -D < myscript.sql).

This query compare tables and get the IDs on dev not present on production. Then, execute a select for the complete table (only these ids), and insert them on prod.

(Replace the id field with your unique identifier one for each table).

Answer 2

This seems like a pretty strange approach. Usually the data in development is regarded as disposable. You want just enough data so that you can do styling and troubleshooting - usually with psuedorandom data. Building the "finished" app data in development seems error prone and you would need to sync work if you are more than one developer.

Plus if the data set is significantly large Rails will be very slow in development due to the lack of caching.

What you want is a staging environment which runs on the same settings as the intended production. The key here is that it should be as close to production as possible. This can run on remote server or a server in a intranet.

You can also use the staging environment to display new features or progress to clients/stakeholders and let them preview new features or be looped in on the progress in development.

You can create it by copying config/environments/production.rb -> staging.rb and by setting the RAILS_ENV env var to staging on the intended staging server.

You should also create an additional section in config/database.yml or use ENV['DATABASE_URL'] .

Depending on the project staging can be flushed daily with mirrored data from production or be fully synced.