how to dump data from one table to another

Question

I have staging table in job in SQL server which dumped data in another table and once data is dumped to transaction table I have truncated staging table.

Now, problem occurs if job is failed then data is transaction table is roll back and all data is placed in staging table. So staging table already consist data and if I rerun the job then it merges all new data with existing data in staging table.

I want my staging table to empty when the job will run. Can I make use of temp table in this scenario?

Answer 1

This is a common scenario in data warehousing project and the answer is logical instead of technical. You've two approaches to deal with these scenario:

1. If your data is important then first check if staging is empty or not. If the table is not empty then it means last job failed; in this case instead of inserting into staging do a insert-update operation and then continue with the job steps. If the table is empty then it means that last job was successful then new data will be insert only.

2. If you can afford data loss from last job then make a habit to truncate table before running your package.