Sql server large import without bulk copy or SSIS

Question

I need to import around 100k records a few dozen times a day into an AWS hosted SQL Server Web 13.00.2164.0.v1

AWS doesn't support bulk insert, and SQL Server Web doesn't support SSIS.

I am reading records from a csv file with a C# console app, performing some data transformations and then inserting 1 record at a time with SqlCommand under a single transaction per 100k record file.

My current rate is around 25k records per 30 minutes, which seems ridiculously slow. I had originally developed this process using bulk inserts and could get 100k records inserted in about a minute. Is there anything I can do to speed this up?

Answer 1

You have to convert your data in xml format and then transfer this xml data in to sql. Like bellow example

getting data from a dataset to xml format. string objStr = ds.GetXml();//ds is a dataset object which contains table data

at database end:

CREATE PROCEDURE [dbo].[procedure name]
    -- Add the parameters for the stored procedure here
    @Val varchar(max) = null

AS
BEGIN
    declare @xml xml
    set @xml = convert(xml,@Val)

    SELECT 
        T.Node.value('colname[1]', 'numeric(18, 2)') AS colname
    FROM   
        @xmlValue.nodes('/NewDataSet/Table0') AS T(Node)
END

Answer 2

The native way to do it is to use Table-Valued Parameters .

See also Table-Valued Parameters in .NET for explanations how to use them in your C# code.

So, create a stored procedure that accepts a table-valued parameter and pass all 100K rows in one call. You may try to experiment with the size of the batch and try smaller batches, but 100K is not too much.

In this stored procedure there will be a single INSERT statement that inserts rows from the parameter table into the permanent table.

It will be definitely faster than inserting one row at a time.