sql异常非常慢……如何加快速度？

Question

I have a piece of code that imports data into a sql table. When there's a problem with the incoming data (wrong data type or would be truncated, for example) it generates an exception as expected, but it's very slow. I could import 10,000 rows in seconds if they're all good, but sometimes the source data needs a little tlc and so the app imports the good rows and reports the exceptions so they can be fixed. If, say, all 10,000 rows have a field that needs to be shortened because it would be truncated then it would take more like 10 minutes instead of just 10 or 20 seconds.

The added time is not because of any code being run in the catch block... it does this even if there is no code in the catch block at all. Anyone know if this is just the way it is or if there's anything that can be done to make it faster?

Edit: Okay, per request I'm adding some code below but I think you will quickly see why I didn't think it was necessary to do so at first :)

    public void ImportRow(DataRow r, SqlConnection conn, SqlTransaction trx)
    {
        var sqlCmd = _importCommand.CreateSqlCommand(r);
        SqlCommand sCom = new SqlCommand(sqlCmd, conn, trx);

        try
        {
            sCom.ExecuteNonQuery();
        }
        catch (Exception e)
        {

        }
    }

_importCommand.CreateSqlCommand(r) just returns a string (as you probably already inferred by it's use in the SqlCommand constructor parameter). And as you can see I have not added any code whatsoever to the catch block yet. When there is no exception it executes sCom.ExecuteNonQuery(); very quickly, but when there is an exception there is a short pause. The ImportRow method is called in a loop from another function, but I have positively identified the lag as coming from the try block of this method so the other code is not relevant.

Answer 1

When you hit an error during a bulk insert operation the successfully inserted records have to be rolled back. As a rule of thumb, the undo of an operation during rollback will always be slower compared to the operation being undone, because it has to read back the log (which is optimized for write, not for read) and compensate the operation. However a different from 10 seconds to 10 minutes cannot be explained by this. Therefore, unless you can provide some sort of evidence that is the server that takes 10 minutes, I must conclude that is you application code that is doing something for those 10 minutes.

Updated

Lets try 10k inserts that all fail and compare them with 10k inserts that succeed:

set nocount on;
use master;
if db_id('test') is not null
begin
    alter database test set single_user with rollback immediate;
    drop database test;
end
go

create database test;
go

use test;
go

create table good (a int, b char(1000));
go

declare @start datetime = getdate(), @i int =0;
begin transaction;
while @i < 10000
begin
    insert into good (a) values (@i);
    set @i += 1;
end
commit;
declare @end datetime = getdate();
select 'good: ', datediff(ms, @start, @end);
go

create table bad (a int, b char(1000), constraint fail check (a<0));
go

declare @start datetime = getdate(), @i int =0;
begin transaction;
while @i < 10000
begin
    insert into bad (a) values (@i);
    set @i += 1;
end
commit;
declare @end datetime = getdate();
select 'bad: ', datediff(ms, @start, @end);
go

On my test machine I get ~600ms for the 'good' case and about 1400ms for the bad. So the exceptions double the time, but nowhere near minutes.

Next, lets do the very same, but from managed client:

    static void Main(string[] args)
    {
        try
        {
            using (SqlConnection conn = new SqlConnection(@"..."))
            {
                conn.Open();
                Stopwatch sw = new Stopwatch();
                sw.Start();
                using (SqlTransaction trn = conn.BeginTransaction())
                {
                    SqlCommand cmd = new SqlCommand(
                       "insert into good (a) values (@i)", conn, trn);
                    SqlParameter p = new SqlParameter("@i", SqlDbType.Int);
                    cmd.Parameters.Add(p);
                    for (int i = 0; i < 10000; ++i)
                    {
                        p.Value = i;
                        cmd.ExecuteNonQuery();
                    }
                    trn.Commit();
                    sw.Stop();
                }
                Console.WriteLine("Good: {0}", sw.Elapsed);

                int excount = 0;
                sw.Reset();
                sw.Start();

                using (SqlTransaction trn = conn.BeginTransaction())
                {
                    SqlCommand cmd = new SqlCommand(
                        "insert into bad (a) values (@i)", conn, trn);
                    SqlParameter p = new SqlParameter("@i", SqlDbType.Int);
                    cmd.Parameters.Add(p);
                    for (int i = 0; i < 10000; ++i)
                    {
                        p.Value = i;
                        try
                        {
                            cmd.ExecuteNonQuery();
                        }
                        catch (SqlException s)
                        {
                            ++excount;
                        }
                    }
                    trn.Commit();
                    sw.Stop();
                }
                Console.WriteLine("Bad: {0} [Exceptions: {1}]",  
                   sw.Elapsed, excount);
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(e);
        }
    }

The results (both Retail and Debug builds have similar times):

Good: 00:00:00.8601303
Bad: 00:01:57.8987760 [Exceptions: 10000]

So the time has gone into nearly 2 minutes from the 1.4 seconds that is the raw SQL Server time. So case closed, exceptions are expensive, right? Not so fast. Try running the application without debugger attached (Ctrl+F5):

Good: 00:00:00.6640281
Bad: 00:00:02.3746845 [Exceptions: 10000]

Back to 2 seconds. so the true culprit is not SQL Server, is not C# exception handling nor the SqlClient layer. Is the Debugger . This is normal, since the debugger does some pretty invasive things and runs a lot of code on each CLR exception thrown. If your code does basically nothing but throw exceptions, the results can have a big impact. It is called First Chance Exception Handling and other have said it before .

However, if the application is being debugged, the debugger sees all exceptions before the program does. This is the distinction between the first and second chance exception: the debugger gets the first chance to see the exception (hence the name). If the debugger allows the program execution to continue and does not handle the exception, the program will see the exception as usual.

Yet again, proof that when dealing with performance issues, best is to measure .

Answer 2

If you perform the same checks on your client, you'll be able to report those find the good records very quickly, and report errors back quickly as well.

You can dynamically deduce the layout of the table you're writing to, as to need duplicated the table layout specifications, but I don't think it's worth the bother unless your component is very generic in nature.

Answer 3

Some tips:

1. Create a parameterised query, prepare it and reuse it. This way, SQL Server can cache the execution plan
2. If you've got a lot of data to import, look into using the bulk import functionality ( MERGE and OPENROWSET are my current favourite for cross platform fun) - they are a lot quicker than using individual update/insert statements.

Answer 4

It seems strange that an exception would be causing so much of a slow-down, unless of course you were rolling back the transaction in which case I can totally understand the slow-down on a problem.

Anyway, one thing you could try is to deduce the length of the columns from the table definition, and truncate the data PRIOR to inserting in to the database. Therefore, you'll only ever been inserting good data.

Answer 5

Exceptions are slow, because while throwing and catching, there is lot of bookkeeping that needs to be done, it's all finalizes that need to execute and all stack frame calculations which takes longer in general anyway. I don't think you can speed up this in anyway at all by using some inbuilt methods.

You will have to run checks by yourself instead of SQL to return you the error. I know it is little tedious but there is no way to speed up.

Another alternative would be to insert rows in parallel threads. This will speed up, as exception in one row will not slow down all pending rows.