Stored procedure to copy only latest records from one table to another

Question

I have a situation where my source table TABLE_A gets frequently updated on hourly basis through some front end consumer app. I've another reporting table TABLE_B which gets some of its data from TABLE_A.

Initially copying of records which were done manually through insert statement. Now this needs to be automated where a stored procedure/script runs every 1 hr and copies the records. So when this script does a 2nd run, it should not copy the records which it has already copied in its first run. Please help me in how I can do this.

TABLE_A:

id  date         product    type
11  2020-10-14    abc        T
12  2020-10-16    def        P
- - - - - - - - - -- - - - -- - - - -- - - -- 
13  2020-10-17    ghi        K
14  2020-10-18    klm        L
15  2020-10-19    abc        T

TABLE_B

id        date              product       type
hadgha     2020-10-14         abc          T
gsggss     2020-10-16         def          P 

Suppose the script has copied the first 2 records in its first run and after that records 13 / 14 / 15 has been added to TABLE_A. Now in the second run the script should copy the last 3 records from TABLE_A to TABLE_B..

Answer 1

Assuming that id is a primary key in both tables, you can use on duplicate key :

insert into table_b values(id, date, product, type)
select id, date, product, type
from table_a
on duplicate key update id = values(id)

Basically this attempts to insert everything from table_a to table_b ; when an already-existing record is met, the query goes to the on duplicate key clause, that performs a "dummy" update (the values are the same already, so that's a no-op: MySQL skips the update).

For this to properly work, I would recommend:

• setting id in table_a to int primary key auto_increment

• setting id in table_b to a int primary key (not auto increment, since values will always be copied from the other table)

Answer 2

I would suggest filtering them out using a subquery and then using on duplicate key :

insert into table_b values(id, date, product, type)
    select id, date, product, type
    from table_a a
    where a.date = (select max(a2.date) from table_a a2 where a2.type = a.type and a2.product = a.product)
    on duplicate key update id = values(id);

This assumes that you have a unique constraint on table_b(product, type) .

Answer 3

The tables don't have any duplicate key or any constraints in common. The stored procedure that is being created has variables to capture the max count id and then updating this max count id on every run.. this way it is ensured that it copies the records that has not been copied in last run.