根据暂存表中的snapshot_date和id将记录更新/插入不同的快照表中

Question

I have a huge snapshot table (example user_snapshot_all ) broken into the different smaller table on Redshift (ie Postgres) to get the performance gain.

So, the smaller tables are like (suffix has year_month)

user_snapshot_1995_1
user_snapshot_1995_2
user_snapshot_1995_3
user_snapshot_1995_4
....
user_snapshot_2016_11

They hold snapshot record for whatever year and month they have suffix

I use a staging table user_snapshot_staging to load/update data to these tables incrementally, in 99% cases, it is just the latest year_month one.

But there will be some edge cases like 12:00 midnight when staging table will have data which will span to two tables (for example user_snaspshot_2016_10 and user_snapshot_2016_11 on 2016-11-1) Or another edge case, maybe we need to update few 2 years old snapshot and so staging table will have some 2-year-old records along with a lot of today's snapshot.

The question is how I should design my query or code so that it can update or insert data into right month_year snapshot table?

All the snapshot tables and staging tables have at least these two columns:

id
snapshot_date

To clarify further: If it was single user_snapshot_all I could easily update the records by joining the staging table with master table based on snapshot_date and id. But with these smaller tables segmented by month_year, there is no guarantee that all records from staging tables can be found in one snapshot table.

Here are use cases Note: Below queries are part of an ETL process, they are not one-off manual one, that is why I need to be automated solution.

Scenario 1) Suppose user_snapshot_staging table has

id  snapshot_date user_detail
100  2016-11-3     jskesljd234
101  2016-11-4     jskesljdfg23
102  2016-11-5     jskesljdbd23
103  2016-11-6     jskesljdw23ds

since all the snapshot belongs to November 2016, all this data will be Inserted/Updated into user_snapshot_2016_11 with following two queries:

Insert new:

Insert into user_info_snapshot_2011_11 (id, snapshot_date, user_detail )
from user_info_snapshot_staging source LEFT OUTER JOIN user_info_snapshot_2011_11 target on source.id = target.id where target.id is null
;

Update existing:

update user_info_snapshot_2011_11 set snapshot_date=source.snapshot_date, user_detail=source.user_detail 
from user_info_snapshot_staging source INNER JOIN user_info_snapshot_2011_11 target on source.id = target.id where

Scenario 2) Now suppose user_snapshot_staging table has

id  snapshot_date user_detail
1300  2015-01-3     jskesljd234
1301  2015-10-4     jskesljdfg23
1302  2016-11-1     jskesljdbd23
1303  2016-11-2     jskesljdw23ds

Now staging table has snapshots which will require update and insert to different snapshot tables, we cannot just insert/update into user_snapshot_2016_11, but we need to also insert/update into user_snapshot_2015_01 and user_snapshot_2015_10

How should I design my query or code which generate the dynamic query to handle these cases so that only appropriate table are joined with user_snapshot_staging table based on data in the staging table?

Let me know if you need further clarifications. Sorry, it is little tricky to explain.

Answer 1

You can generate queries with the following approach. I'll give examples in pseudo-code based on python syntax.

1. Get year/month combinations you have in your staging database

SELECT DISTINCT to_char(date, 'YYYY-MM') FROM user_info_snapshot_staging;

2. These are your query templates:

-- insert_template.sql
INSERT INTO user_info_snapshot_{{ year }}_{{ month }} (id, snapshot_date, user_detail )
FROM user_info_snapshot_staging source LEFT OUTER JOIN user_info_snapshot_{{ year }}_{{ month }} target on source.id = target.id where target.id is null
WHERE DATE_TRUNC('month', source.date) = {{ month }} AND DATE_TRUNC('year', source.date) = {{ year }};

-- update_template.sql
UPDATE user_info_snapshot_{{ year }}_{{ month }} SET snapshot_date=source.snapshot_date, user_detail=source.user_detail
FROM user_info_snapshot_staging source INNER JOIN user_info_snapshot_{{ year }}_{{ month }} target on source.id = target.id where
DATE_TRUNC('month', source.date) = {{ month }} AND DATE_TRUNC('year', source.date) = {{ year }};

Now loop through year/month pairs and execute those queries:

for year_month, in cursor.execute("SELECT to_char('YYYY-MM', date_columns) FROM user_info_snapshot_staging"):
    year, month = year_month.split('-')
    # this is where you generate sql
    sql = template('insert_template', context={
        'year': year,
        'month': month,
    })
    # here you execute it
    cursor.execute(sql)

I would advise against using update if you need to update a lot of records. Further info in this question .