在插入期间避免重复

Question

I am working on a stored procedure that currently builds our fact table every hour. Currently, during hourly refresh it truncates the table and Inserts new data every time. I am trying to change that to only delete rows that are not needed and append new rows. I have gotten the delete part, but currently, as the ID column (Primary Key) is created upon Insertion, I am not sure how to avoid the insertion of duplicate records, which is what I am currently seeing.

Currently, the stored procedure inserts the primary key (ID) upon insert. I've taken out the truncate table query and replaced that with a delete query. Now I need to work on avoiding duplicates during the insert.

   --INSERT DATA FROM TEMP TABLE TO FACTBP
   INSERT INTO dbo.FactBP
   SELECT 
   [SOURCE]
  ,[DC_ORDER_NUMBER]
  ,[CUSTOMER_PURCHASE_ORDER_ID]
  ,[BILL_TO]
  ,[CUSTOMER_MASTER_RECORD_TYPE]
  ,[SHIP_TO]
  ,[CUSTOMER_NAME]
  ,[SALES_ORDER]
  ,[ORDER_CARRIER]
  ,[CARRIER_SERVICE_ID]
  ,[CREATE_DATE]
  ,[CREATE_TIME]
  ,[ALLOCATION_DATE]
  ,[REQUESTED_SHIP_DATE]
  ,[ADJ_REQ_SHIP]
  ,[CANCEL_DATE]
  ,[DISPATCH_DATE]
  ,[RELEASED_DATE]
  ,[RELEASED_TIME]
  ,[PRIORITY_ORDER]
  ,[SHIPPING_LOAD_NUMBER]
  ,[ORDER_HDR_STATUS]
  ,[ORDER_STATUS]
  ,[DELIVERY_NUMBER]
  ,[DCMS_ORDER_TYPE]
  ,[ORDER_TYPE]
  ,[MATERIAL]
  ,[QUALITY]
  ,[MERCHANDISE_SIZE_1]
  ,[SPECIAL_PROCESS_CODE_1]
  ,[SPECIAL_PROCESS_CODE_2]
  ,[SPECIAL_PROCESS_CODE_3]
  ,[DIVISION]
  ,[DIVISION_DESC]
  ,[ORDER_QTY]
  ,[ORDER_SELECTED_QTY]
  ,[CARTON_PARCEL_ID]
  ,[CARTON_ID]
  ,[SHIP_DATE]
  ,[SHIP_TIME]
  ,[PACKED_DATE]
  ,[PACKED_TIME]
  ,[ADJ_PACKED_DATE]
  ,[FULL_CASE_PULL_STATUS]
  ,[CARRIER_ID]
  ,[TRAILER_ID]
  ,[WAVE_NUMBER]
  ,[DISPATCH_RELEASE_PRIORITY]
  ,[CARTON_TOTE_COUNT]
  ,[PICK_PACK_METHOD]
  ,[RELEASED_QTY]
  ,[SHIP_QTY]
  ,[MERCHANDISE_STYLE]
  ,[PICK_WAREHOUSE]
  ,[PICK_AREA]
  ,[PICK_ZONE]
  ,[PICK_AISLE]
  ,EST_DEL_DATE
  ,null
  --,[ID]
  FROM #TEMP_FACT
  --code for avoiding duplicates

   --CLEAR ALL DATA FROM FACTBP
   DELETE FROM dbo.FactBP
   WHERE SHIP_DATE < DATEADD(s,-1,DATEADD(mm, 
   DATEDIFF(m,0,GETDATE())-2,0)) and SHIP_DATE IS NOT NULL

Answer 1

You need to check against the natural key . Since you're talking about a fact table, the natural key is probably the combination of a lot of fields. If we assume SOURCE and DC_ORDER_NUMBER make up the natural key, this should work:

INSERT INTO dbo.FactBP

SELECT 
  t.[SOURCE]
, t.[DC_ORDER_NUMBER]
, t.[CUSTOMER_PURCHASE_ORDER_ID]
, t.[BILL_TO]
, t.[CUSTOMER_MASTER_RECORD_TYPE]
, t.[SHIP_TO]
, t.[CUSTOMER_NAME]
, t.[SALES_ORDER]
, t.[ORDER_CARRIER]
, t.[CARRIER_SERVICE_ID]
, t.[CREATE_DATE]
, t.[CREATE_TIME]
, t.[ALLOCATION_DATE]
, t.[REQUESTED_SHIP_DATE]
, t.[ADJ_REQ_SHIP]
, t.[CANCEL_DATE]
, t.[DISPATCH_DATE]
, t.[RELEASED_DATE]
, t.[RELEASED_TIME]
, t.[PRIORITY_ORDER]
, t.[SHIPPING_LOAD_NUMBER]
, t.[ORDER_HDR_STATUS]
, t.[ORDER_STATUS]
, t.[DELIVERY_NUMBER]
, t.[DCMS_ORDER_TYPE]
, t.[ORDER_TYPE]
, t.[MATERIAL]
, t.[QUALITY]
, t.[MERCHANDISE_SIZE_1]
, t.[SPECIAL_PROCESS_CODE_1]
, t.[SPECIAL_PROCESS_CODE_2]
, t.[SPECIAL_PROCESS_CODE_3]
, t.[DIVISION]
, t.[DIVISION_DESC]
, t.[ORDER_QTY]
, t.[ORDER_SELECTED_QTY]
, t.[CARTON_PARCEL_ID]
, t.[CARTON_ID]
, t.[SHIP_DATE]
, t.[SHIP_TIME]
, t.[PACKED_DATE]
, t.[PACKED_TIME]
, t.[ADJ_PACKED_DATE]
, t.[FULL_CASE_PULL_STATUS]
, t.[CARRIER_ID]
, t.[TRAILER_ID]
, t.[WAVE_NUMBER]
, t.[DISPATCH_RELEASE_PRIORITY]
, t.[CARTON_TOTE_COUNT]
, t.[PICK_PACK_METHOD]
, t.[RELEASED_QTY]
, t.[SHIP_QTY]
, t.[MERCHANDISE_STYLE]
, t.[PICK_WAREHOUSE]
, t.[PICK_AREA]
, t.[PICK_ZONE]
, t.[PICK_AISLE]
, t.EST_DEL_DATE
, null
--,[ID]

FROM #TEMP_FACT t
  left outer join dbo.FactBP f on f.[SOURCE] = t.[SOURCE]
                              and f.[DC_ORDER_NUMBER] = t.[DC_ORDER_NUMBER]

where f.[SOURCE] is null

Adjust the join and the WHERE clause to match the natural key of the table.

You should also take another look at your DELETE script. Do you really want to delete all records with a SHIP_DATE < 2019-07-31 23:59:59.000 ? Or should that be <= ? Maybe this will work better (and simpler):

DELETE FROM dbo.FactBP
WHERE SHIP_DATE < cast(dateadd(day, 1, EOMONTH(getdate(), -3)) as datetime2)
  and SHIP_DATE IS NOT NULL