How can I speed up aggregate SQL insertions?

Question

I need to update 2 tables of a MS SQL Server DB.

For each "Event" I generate 2 queries:

1. Insertion in table A
2. Update of related row in table B

So for each Event I have a query like:

cmdA = "INSERT INTO [TableA] very complex referncing TableB"
cmdB = "UPDATE [TableB] SET very complex referencing TableA"

Because the queries reference each other, I can't do a sort of table upload approach.

I have an Event queue that gets filled by another thread and I need to execute cmdA and cmdB as quickly as possible. Running through the queue event by event is too slow. So, I create an aggregate command by taking all current queue Events and concatenateing them with ";"

aggCmdTxt = cmdA;cmdB;cmdA;cmdB;cmdA;cmdB;cmdA;cmdB...

I then execute this aggregate command using InsertOrdersFromTxt below.

This is MUCH faster than running cmdA & cmdB for each event, BUT is still slow. If I get 30 Events in my queue it can take 500 ms.

It appears that the number of sub-queries in my aggregate command proportionally affects the execution time. This I would of course expect - but can I make it more efficient?

public void InsertOrdersFromTxt(string aggCmdText)
{
    SqlCommand cmd;

    try
    {
        cmd = new SqlCommand(aggCmdText, conn);

        cmd.ExecuteNonQuery();
    }
    catch (Exception err)
    {
        throw new Exception("[InsertOrdersFromTxt] Error: " + err.Message);
    }
}

Answer 1

Using set based approach is the best approach for the SQL Server. So try to use the following approach.

If you want to update data Table B based on some conditions of Table A , then you can send just data into a stored procedure which can update data of Table B :

C#:

// creating table-valued parameter
var yourDataTable = new DataTable();
yourDataTable.Columns.Add("ID", typeof(int));
yourDataTable.Columns.Add("CarNumber");

//inserting data into table valued parameter
yourDataTable.Rows.Add(1, "Hello, World!");

//Parameter declaration
SqlParameter[] Parameter = new SqlParameter[1];    
Parameter[0].ParameterName = "@FooParameters";    
Parameter[0].SqlDbType = SqlDbType.Structured;    
Parameter[0].Value = yourDataTable;    

// Executing Stored Procedure
SqlHelper.ExecuteNonQuery(this.ConnectionString, CommandType.StoredProcedure, " 
    [YourStoredProcedure]", Parameter);  

Then create table valued parameter at SQL Server:

CREATE TYPE [dbo].[tp_YourTP] AS TABLE(
    [ID] [INT] NOT NULL,
    [CarNumber] [VARCHAR](255) NULL
)
GO

And then you in your stored procedure you can use this table parameter:

DROP PROCEDURE IF EXISTS YourStoredProcedure
GO 

CREATE PROCEDURE dbo.YourStoredProcedure(     
      @FooParams  tp_YourTP  READONLY        
)
AS
   UPDATE t_A
   SET t_A.CarNumber = t_B.CarNumber
   FROM TableA AS t_A
   INNER JOIN @FooParams  AS t_B ON t_A.ID = t_B.ID

Answer 2

Please check this :

using (connection)
{
    SqlCommand command = new SqlCommand(
      "SELECT CategoryID, CategoryName FROM dbo.Categories;" +
      "SELECT EmployeeID, LastName FROM dbo.Employees",
      connection);
    connection.Open();

    SqlDataReader reader = command.ExecuteReader();

    while (reader.HasRows)
    {
        Console.WriteLine("\t{0}\t{1}", reader.GetName(0),
            reader.GetName(1));

        while (reader.Read())
        {
            Console.WriteLine("\t{0}\t{1}", reader.GetInt32(0),
                reader.GetString(1));
        }
        reader.NextResult();
    }
}

Answer 3

One thin is that update is much slower then insert. So,

Solution 1: Bulk all the insert statement into one statement and execute it. For the update read the rows base on PK from table and delete all the rows from the table. Then execute insert statement for all the rows instead update stm.

Solution 2: Use bulk insert using SqlBulkCopy class. ex; https://www.csvreader.com/code/cs/bulk_insert_csv.php .

Solution 3: Using Asynchronous programming in C# execute your statement in bulk. This way your main thread will not stop by sql operation to be completed.

Solution 4: Create connection pool. Use multi thread to update update your table A and B separately.

It is possible to combined the solution to get better result.