Sync 2 tables: one with engine = Memory another with engine = InnoDB

Question

I have 2 tables, student_memory (engine = Memory) and student_innodb (engine = InnoDB). student_memory is being updated continuously (at an interval of 2 seconds) by some thread. I have to sync both the tables (say at an interval of 10 seconds). I have some methods for that:
1. Create insert/update/delete statements by seeing the difference between the 2 tables and run those query on student_innodb.
2. Drop student_innodb , ALTER TABLE student_memory ENGINE = INNODB, RENAME TO student_innodb ;
3. Truncate student_innodb table and do, insert into student_innodb select * from student_memory ;

I'm using 2nd approach as this is saving me from comparing rows of the tables, creating INSERT/UPDATE/DELETE and executing them. But I'm not sure about the performance. There can be around 1000-100000 Rows in these tables. Can anyone suggest any other better solution or which one should I use?

Answer 1

To answer your question, possibly the fastest way to do this is to have some sort of primary key on both tables (eg student_id). You should then add a third memory table changed_students_memory that has only a student_id field as it's primary key.

Every time you make a change to students_table, you should:

INSERT (student_id) VALUES (:student_id) INTO changed_students_memory ON DUPLICATE KEY IGNORE

Then you have just a list of records to update in changed_students_memory. You can then update your innodb table as such:

BEGIN;

UPDATE
  student_innodb AS i
  JOIN student_memory AS m USING (student_id)
  JOIN changed_students_memory USING (student_id)
SET
  i.<fieldname1> = m.<fieldname1>,
  i.<fieldname2> = m.<fieldname2>... ;

TRUNCATE TABLE changed_students_memory;

COMMIT;

This will be much faster if you're only updating a small percentage of the rows. I'd also mention that under that assumption using the third table is preferable to adding a "dirty" column to the exiting memory table and adding an index on it, since most times the optimizer won't use the index since it won't think it's selective enough.

Having said all of that - I'm guessing that if you're doing all this craziness with an in-memory table, you're approaching your problem wrong. In-memory tables are not meant to store data for long periods of time. If the database goes down, you will lose your data. If you have in-memory table as a performance optimization, you're better off tweaking your INNODB setup instead. INNODB is very fast if configured correctly. I have it easily handling load from hundreds of concurrent connections. INNODB will keep all hot data in memory if you give it enough memory (see innodb_buffer_pool_size in my.cfg/my.ini).

Answer 2

I know that this is an old question, but for people that come from search engines (like myself) I should say:

NEVER implement any of solutions in above question, by first one you could get partially data lost and by two other solutions, you could get full data lost!

The best way is INSERT/UPDATE/DELETE data on innodb(or memory), and set AFTER INSERT/AFTER UPDATE/AFTER DELETE triggers to update memory the other table. So, you never lose data, but don't forget to enable Transaction for INSERT/UPDATE/DELETE!

This way, you never get data loss, and it is very fast to read data from memory table. And Memory table will always get synced to Innodb one automatically.

1 Last thing: don't forget to config mysql to automatically sync this two tables at server start!