如何确保将所有行都插入到数据模型中，然后截断临时表

Question

In my ETL job I would like to truncate stg table but before truncating stging table, I want to make sure that all the records are inserted in the data model. The sample is as below

create table #stg (
    CreateID int,
    Name nvarchar(10)
)
insert into #stg
select 1, 'a' union all
select 2, 'b' union all
select 3, 'c' union all
select 4, 'd' 

go

create table #Approve1 (
    CreateID int,
    Name nvarchar(10)
)
insert into #Approve1 
select 1, 'a' union all
select 4, 'd' 

create table #Approve2 (
    CreateID int,
    Name nvarchar(10)
)
insert into #Approve2
select 3, 'c'

create table #Reject (
    CreateID int,
    Name nvarchar(10)
)
insert into #Reject
select 2, 'b'



select * from #stg
select * from #Approve1
select * from #Approve2
select * from #Reject`

How can I check among these tables and make sure that all values are loaded into the data model and the staging table can be truncated.

Thanks

Answer 1

You can check that the values exist in at least one of the tables. Here is a method that counts the number of records remaining in the staging table using not exists :

select count(*)
from #stg s
where not exists (select 1 from #Approve1 a where a.CreateId = s.CreateId and a.name = s.name) and
      not exists (select 1 from #Approve2 a where a.CreateId = s.CreateId and a.name = s.name) and
      not exists (select 1 from #Reject r where r.CreateId = s.CreateId and r.name = s.name);

If this returns a value greater than 0 , then not everything was inserted. (And by using select * you can see what is missing.)

Of course, this doesn't guarantee what you want, because the values could have existed in the table earlier. I assume this is a simplified table structure and you have some mechanism for identifying the most recently inserted records.