使用大型Excel文件插入/更新SQL

Question

I have a situation where I have to insert/update a lot of information in a table for our ERP. I first have to check the table to see if the information exists, if not, insert, if so, update.

I have sample code below.

    IF NOT EXISTS(SELECT * FROM EXT00101 WHERE PT_Window_ID='ITEM_SHIP_MAINT' and PT_UD_Key='18 RND PA' and PT_UD_Number=5)
    BEGIN
          INSERT INTO EXT00101 VALUES('ITEM_SHIP_MAINT', '18 RND PA', 5, '70')
    End

IF EXISTS(SELECT * FROM EXT00101 WHERE PT_Window_ID='ITEM_SHIP_MAINT' and PT_UD_Key='18 RND PA' and PT_UD_Number=5)
    BEGIN
        UPDATE EXT00101
        SET STRGA255='70'
        WHERE PT_Window_ID='ITEM_SHIP_MAINT' and PT_UD_Key='18 RND PA' and PT_UD_Number=5;
    END

I used a mail merge against an excel file which has 23,000+ lines. This causes my SQL statement to be 260,000+ lines. There has to be a more efficient way to complete this.

If not, I will have to break this code up considerably.

Microsoft SQL Server 2005

Answer 1

To get you on the right tracks using SSIS, the first step would be to use an Execute SQL task to create a staging table:

IF (OBJECT_ID(N'dbo.tmpEXT00101Staging') IS NOT NULL 
    DROP TABLE dbo.tmpEXT00101Staging;

CREATE TABLE dbo.tmpEXT00101Staging
(       PT_Window_ID    VARCHAR(50) NOT NULL,
        PT_UD_Key       VARCHAR(50) NOT NULL,
        PT_UD_Number    INT NOT NULL,
        STRGA255        VARCHAR(50) NOT NULL
);

Then use a Data Flow task to import your data to this table (flat file or excel source, and your destination would be OLE DB Destination. You may need to set "Delay Validation" on the Data Flow task to false and/or "Validate External Meta Data" to false on the destination because the destination is created at run time). You could also just keep your staging table there permanently, and instead just use TRUNCATE TABLE dbo.tmpEXT00101Staging to clear it at the start of each execution.

Finally use this staging table to update your main table (and drop the staging table to clean up if you don't want to keep it permanently)

BEGIN TRAN;

-- UPDATE ROWS THAT EXIST
UPDATE  t
SET     STRGA255 = st.STRGA255
FROM    dbo.EXT00101 t
        INNER JOIN dbo.tmpEXT00101Staging st
            ON st.PT_Window_ID = T.PT_Window_ID
            AND st.PT_UD_Key = t.PT_UD_Key
            AND st.PT_UD_Number = t.PT_UD_Number;

-- INSERT ROWS THAT DO NOT EXIST
INSERT EXT00101 (PT_Window_ID, PT_UD_Key, PT_UD_Number, STRGA255)
SELECT PT_Window_ID, PT_UD_Key, PT_UD_Number, STRGA255
FROM    dbo.tmpEXT00101Staging st
WHERE   NOT EXISTS
        (   SELECT  1
            FROM    dbo.EXT00101 t
            WHERE   st.PT_Window_ID = T.PT_Window_ID
            AND     st.PT_UD_Key = t.PT_UD_Key
            AND     st.PT_UD_Number = t.PT_UD_Number
        );

COMMIT TRAN;

-- CLEAN UP AND DROP STAGING TABLE (OPTIONAL)
IF (OBJECT_ID(N'dbo.tmpEXT00101Staging') IS NOT NULL 
DROP TABLE dbo.tmpEXT00101Staging;

There is potential to meet a race condition here, so you should ensure that you have constraints in to stop any integrity violation of this by concurrent threads trying to insert the same record.

For the sake of completeness, the preferable option to do the UPSERT is MERGE (if you upgrade from 2005 at anypoint):

MERGE dbo.EXT00101 WITH (HOLDLOCK) AS t
USING dbo.tmpEXT00101Staging AS st
    ON st.PT_Window_ID = T.PT_Window_ID
    AND st.PT_UD_Key = t.PT_UD_Key
    AND st.PT_UD_Number = t.PT_UD_Number
WHEN MATCHED THEN 
    UPDATE  
    SET STRGA255 = st.STRGA255
WHEN NOT MATCHED THEN 
    INSERT (PT_Window_ID, PT_UD_Key, PT_UD_Number, STRGA255)
    VALUES (st.PT_Window_ID, st.PT_UD_Key, st.PT_UD_Number, st.STRGA255);