Replacing a 3 level deep nested cursors in SQL Server

Question

I have three SQL Server tables that I need to loop trhough and update. I did it successfully with a cursor but it is so slow that it is pretty pointless sincethe main table with all the data to loop through is over 1,000 rows long.

The tables are (with some sample data):

-- The PK is InvoiceId and the IsMajorPart is '0' or '1'.
-- The MajorPartId and SubPartId1 to 4 are "technically" FKs for PartId but aren't hooked up and will not be ever due to some external issues outside of scope.
-- The part Id's can be NULL or empty.
-- This table exists elsewhere and is loaded with Id's being varchars but in transfering they will be going in as int's which is the proper way.
CREATE TABLE dbo.Invoices(
    InvoicdeId varchar(50),
    PartName varchar(255),
    IsMajorPart varchar(1)
    MajorPartId varchar(50),
    SubPartId1 varchar(50),
    SubPartId2 varchar(50),
    SubPartId3 varchar(50),
    SubPartId4 varchar(50));

-- Sampe inserts
INSERT INTO dbo.Invoices VALUES ('1', 'A Part', '0', '', '100', '105', '' ,''):
INSERT INTO dbo.Invoices VALUES ('5', 'E Part', '1', '101', '110', '', '' ,''):
INSERT INTO dbo.Invoices VALUES ('11', 'Z Part', '1', '201', '100', '115', '' ,''):


-- Essentially the old table above is being moved into a normalized, correct tables below.


- The PK is the PartId
CREATE TABLE dbo.Parts
    PartsId int,
    PartName varchar(255)

-- Sampe inserts (that will be updated or inserted by looping through the first table)
INSERT INTO dbo.Parts VALUES (100,'A Part'):
INSERT INTO dbo.Parts VALUES (110,'B Part'):
INSERT INTO dbo.Parts VALUES (201,'C Part'):


-- The PK is the combination of InvoiceId and PartId
CREATE TABLE dbo.InvoiceToParts
    InvoiceId int,
    PartsId int,
    IsMajorPart bit);

-- Sampe inserts (that will be inserted from the first table but conflicts might occur if an InvoiceId from the first table has 2 PartId's that are the same)
INSERT INTO dbo.Parts VALUES (1, 100, 0):
INSERT INTO dbo.Parts VALUES (5, 100, 1):
INSERT INTO dbo.Parts VALUES (17, 201, 0):

The sample INSERTs above are just samples of the data for seeing what is in the tables.

The rules to move Invoices (I don't care what happens to this table), into the correct tables of Parts and InvoiceToParts are below (and these last two tables are the only ones that I care about.

1. Loop through Invoices and get all the data.
2. First, find out if IsMajorPart is '1' and then get the MajorPartId .
3. Push the MajorPartId with PartName in Parts table if it DOESN'T already exist.
4. Next check InvoiceToParts to see if the PK of InvoiceId and PartId exist.
5. If they do, update IsMaorPart to '1'.
6. If they don't exist, INSERT it.
7. Next do the same process for all SubPartId1 to SubPartId4 .

I have a nested 3-level cursor which performance-wise ran for over 30min before I stopped it as it wasn't even close to finishing and was sucking up all the resources. I am trying to look for a faster way to do this. The Invoices table can have up to about 5,000 rows in it.

Answer 1

You need to unpivot your data and then just do what is called an UPSERT, which has two steps:

1. If exists, update record(s)
2. If not exists, insert record(s)

Plenty of examples if you search for examples online for UPSERT

Table Setup

DROP TABLE IF EXISTS #Invoice
DROP TABLE IF EXISTS #Unpivot
DROP TABLE IF EXISTS #InvoiceToParts
DROP TABLE IF EXISTS #Parts


CREATE TABLE #Parts(
    PartsId int,
    PartName varchar(255)
)

CREATE TABLE #InvoiceToParts(
    InvoiceId int,
    PartsId int,
    IsMajorPart bit
);

CREATE TABLE #Invoice(
    InvoiceId varchar(50),
    PartName varchar(255),
    IsMajorPart varchar(1),
    MajorPartsID varchar(50),
    SubPartsID1 varchar(50),
    SubPartsID2 varchar(50),
    SubPartsID3 varchar(50),
    SubPartsID4 varchar(50)
);

INSERT INTO #Invoice
VALUES ('1', 'A Part', '0', '', '100', '105', '' ,'')
,('5', 'E Part', '1', '101', '110', '', '' ,'')
,('11', 'Z Part', '1', '201', '100', '115', '' ,'')

SQL to Process Data

Will first unpivot the data, then load into Parts table first so the ID's can be referenced before inserting into the junction table InvoicetoParts

SELECT A.InvoiceId
    ,B.*
INTO #Unpivot
FROM #Invoice AS A
CROSS APPLY (
    VALUES 
    (NULLIF(MajorPartsID,''),PartName,IsMajorPart)
    ,(NULLIF(SubPartsID1,''),NULL,0)
    ,(NULLIF(SubPartsID2,''),NULL,0)
    ,(NULLIF(SubPartsID3,''),NULL,0)
    ,(NULLIF(SubPartsID4,''),NULL,0)
) AS B(PartsID,PartName,IsMajorPart)
WHERE B.PartsID IS NOT NULL /*If not data, filter out*/

/*INSERT into table Parts if not exists*/
INSERT INTO #Parts
SELECT PartsID,PartName
FROM #Unpivot AS A
WHERE A.IsMajorPart = 1
AND NOT EXISTS (
    SELECT *
    FROM #Parts AS DTA
    WHERE A.PartsID = DTA.PartsID
    )
GROUP BY PartsID,PartName

/*UPSERT into table dbo.InvoiceParts*/
UPDATE #InvoiceToParts
SET IsMajorPart = B.IsMajorPart
FROM #InvoiceToParts AS A
INNER JOIN #Unpivot AS B
     ON A.InvoiceId = B.InvoiceId
     AND A.PartsId = B.PartsID

INSERT INTO #InvoiceToParts(InvoiceId,PartsId,IsMajorPart)
SELECT InvoiceId
    ,PartsId
    ,IsMajorPart
FROM #Unpivot AS A
WHERE NOT EXISTS (
    SELECT *
    FROM #InvoiceToParts AS DTA
    WHERE A.InvoiceId = DTA.InvoiceID
        AND A.PartsID = DTA.PartsID
    )

SELECT *
FROM #InvoiceToParts

SELECT *
FROM #Parts