Update tables in one database from multiple tables in another database regularly

Question

I have 2 databases in SQL Server, DB1 has multiple tables and some of the tables are updated with new records continuously. DB2 has only 1 table which should contain all the combined info from the multiple tables in DB1, and needs to be updated every 2 hours.

For example, DB1 has 3 tables: "ProductInfo", "StationRecord", "StationInfo". The first 2 tables both have a timestamp column that indicates when a record is created (ie the two tables are updated asynchronously, ONLY when a product passes all stations in "StationRecord" will "ProductInfo" be updated with a new product), and the last table is fixed.

The tables are as follows:

USE DB1
GO

CREATE TABLE ProductInfo 
ProductID bigint Primary Key NOT NULL
TimeCreated datetime    
ProductName nvarchar(255)

CREATE TABLE StationRecord 
RecordID bigint Primary Key NOT NULL
TimeCreated datetime      
ProductID bigint NOT NULL
StationID bigint

CREATE TABLE StationInfo
StationID bigint Primary Key NOT NULL
BOM_used nvarchar(255)

DB2 has only 1 table which contains a composite PK of "ProductID" & "StationID", as follows:

CREATE TABLE DB2.BOMHistory AS
SELECT 
    DB1.ProductInfo.ProductID
    DB1.ProductInfo.TimeCreated AS ProductCreated
    DB1.StationInfo.StationID
    DB1.StationInfo.BOM_used
    FROM DB1.ProductInfo
JOIN DB1.StationRecord
ON DB1.ProductInfo.ProductID = DB1.StationRecord.ProductID
JOIN DB1.StationInfo
ON DB1.StationRecord.StationID = DB1.StationInfo.StationID 
constraint PK_BOMHistory Primary Key (ProductID,StationID)

I figured out the timing portion which is to use create a job with some pre-set schedules, and the job is to execute a stored procedure. The problem is how to write the stored procedure properly, which has to do the following things:

1. wait for the last product to pass all stations (and the "stationInfo" table is updated fully)
2. find all NEW records generated in this cycle in the tables in DB1
3. combine the information of the 3 tables in DB1
4. insert the combined info into DB2.BOMHistory

Here's my code:

ALTER Procedure BOMHistory_Proc
BEGIN
    SELECT
        DB1.ProductInfo.ProductID,
        DB1.ProductInfo.TimeCreated AS ProductCreated
        DB1.StationInfo.StationID,
                DB1.StationInfo.BOM_used
    into #temp_BOMList
    FROM DB1.ProductInfo
        JOIN DB1.StationRecord
        ON DB1.ProductInfo.ProductID = DB1.StationRecord.ProductID
        JOIN DB1.StationInfo
        ON DB1.StationRecord.StationID = DB1.StationInfo.StationID 
    ORDER BY ProductInfo.ProductID
END

SELECT * from #temp_BOMList

INSERT INTO DB2.BOMHistory(ProductID, ProductCreated, StationID, BOM_used)
SELECT  DISTINCT (ProductID, stationID)
FROM #temp_BOMList
WHERE (ProductID, stationID) NOT IN (SELECT ProductID, stationID FROM DB2.BOMHistory)

The Condition in the INSERT statement is not working, please provide some advice.

Also, should I use a table variable or a temp table for this application?

Answer 1

Try:

INSERT INTO DB2.BOMHistory(ProductID, ProductCreated, StationID, BOM_used)
SELECT DISTINCT tb.ProductID, tb.ProductCreated, tb.StationID, tb.BOM_used
FROM #temp_BOMList tb
WHERE NOT EXISTS
(SELECT * FROM DB2.BOMHistory WHERE ProductID = tb.ProductID AND StationID = tb.StationID)