Copy lots of table records from SQL Server Dev to Prod

Question

I have a table with about 5 million records and I only need to move the last 1 million to production (as the other 4 million are there). What is the best way to do this so I don't have to recopy the entire table each time?

Answer 1

Edit: (Sorry for revising so much. I'm understanding your question better now)

Insert into TblProd
     Select * from TblDev where 
     pkey not in (select pkey from tblprod)

This should only copy over records that aren't already in your target table.

Answer 2

A little faster will probably be:

Insert into prod.dbo.table (column1, column2....)
Select column1, column2.... from dev.dbo.table d
where not exists (
    select 1 from prod.dbo.table pc where pc.pkey = d.pkey
    )

But you need to tell us if these tables are on the same server or not

Also how frequently is this run and how robust does it need to be? There are alternative solutions depending on your requirements.

Given this late arriving gem from the OP: no need to compare as I know the IDs > X , then you do not have to do an expensive comparison. You can just use

Insert into prod.dbo.table (column1, column2....)
Select column1, column2.... from dev.dbo.table d
where ID > x

This will be far more efficient as you are only transferring the rows you need.

Answer 3

Since they are on a separate server that changes everything. In short: in order to know what isn't in dev you need to compare everything in DEV to everything in PROD anyway, so there is no simple way to avoid comparing huge datasets.

Some different strategies used for replication between PROD and DEV systems:

A. Backup and restore the whole database across and apply scripts afterwards to clean it up

B. Implement triggers in the PROD database that record the changes then copy only changed records accross

C. Identify some kind of partition or set of records that you know don't change (ie 12 months ago), and only refresh those that aren't in that dataset.

D. Copy ALL of prod into a staging table on the DEV server using SSIS. Use a very similar query to above to only insert new records across the database. Delete the staging table.

E. You might be able to find a third party SSIS component that does this efficiently. Out of the box, SSIS is inefficient at comparative updates.

Do you actually have an idea of what those last million records are? ie are the for a location or a date or something? Can you write a select to identify them?

Based on this comment:

no need to compare as I know the IDs > X will work

You can run this on the DEV server, assuming you have created a linked server called PRODSERVER on the DEV server

INSERT INTO DB.dbo.YOURTABLE (COL1, COL2, COL3...)
SELECT COL1, COL2, COL3...
FROM PRODSERVER.DB.dbo.YOURTABLE
WHERE ID > X

Look up 'SQL Server Linked Servers' for more information on how to create one.

This is fine for a one off but if you do this regularly you might want to make something more robust.

For example you could create a script that exports the data using BCP.EXE to a file, copies it across to DEV and imports it again. This is more reliable as it does it in one batch rather than requiring a network connection the whole time.

Answer 4

If the tables are on the same server, you can do something like this

I am using MySQL, so may be the syntax will be a little bit different, but in my opinion everything should be the same.

INSERT INTO newTable (columnsYouWantToCopy)
    SELECT columnsYouWantToCopy
    FROM oldTable WHERE clauseWhichGivesYouOnlyRecodsYouNeed

If on another server, you can do something like this: http://dev.mysql.com/doc/refman/5.0/en/select-into.html