How can I update a database table with a .Net DataTable and ignore existing records?

Question

I have a .Net DataTable that contains records, all of which are "added" records. The corresponding table in the database may contain millions of rows. If I attempt to simply call the "Update" method on my SqlDataAdapter, any existing records cause an exception to be raised due to a violation of the primary key constraint. I considered loading all of the physical table's records into a second DataTable instance, merging the two, and then calling the Update method on the second DataTable. This actually works exactly like I want. However, my concern is that if there are 30 billion records in the physical table, loading all of that data into a DataTable in memory could be an issue.

I considered selecting a sub-set of data from the physical table and proceeding as described above, but the construction of the sub-query has proved to be very involved and very tedious. You see, I am not working with a single known table. I am working with a DataSet that contains several hundred DataTables. Each of the DataTables maps to its own physical table. The name and schema of the tables are not known at compile time. This has to all be done at run time.

I have played with the SqlBulkCopy class but have the same issue - duplicate records raise an exception.

I don't want to have to dynamically construct queries for each table at run time. If that is the only way, so be it, but I just can't help but think that there must be a simpler solution using what Ado.Net provides.

Answer 1

you could create your insertcommand like this:

declare @pk int = 1
declare @txt nvarchar(100) = 'nothing'
insert into #temp (id, txt)
    select distinct @pk, @txt 
        where not exists (select id from #temp x where x.id = @pk)

assuming that your table #temp (temporary table used for this example) is created like this (with primary key on id)

create table #temp (id int not null, txt nvarchar(100))