What is the best way to maintain consistency in database while migrating it from one to another

Question

Currently I'm trying to do is migrating data from a database let say Database A to Database B I'm doing this using ETL jobs (PENTAHO KJB). both DB are not identical however pk of Database A are stored in Database B

there some job x which is keep on updating Database A with new data. I need to add data in database B' s tables whenever there is any new data in database A.

what approach I'm currently using.

1. maintaining table which have details last updated Database B table times so can compare the same with Database A tables and check with created date of row.

2 check the uniqueness on the basis of PK if the PK is of table A is present in table B that means this data is present in table B.

what will be the best way to do it. If any of my approach is correct?

Answer 1

Lots of ways you can go about engineering something like this. What you're doing sounds fine, and if it's not broken, don't fix it.

That said, I think the Dimension Lookup/update step in Pentaho might work well for you. You can specify a technical key in the step and then Update or Insert (in your case, insert) all the data you want from one table to another. It's originally meant for slowly changing dimensions, but I think it'll work in your case too. You might check it out. Although, depending on the size of these tables you're updating, this step might be too slow.

https://wiki.pentaho.com/display/EAI/Dimension+Lookup-Update