SQL Server插入超时已过期

Question

I am trying to insert 200,000 documents from a folder into the SQL Server database into a varbinary column. I get a timeout expiration message after inserting 80,000 documents. Average file size is about 250kb and max file size is 50MB. I am running this c# program on the server where the database is located.

Please suggest.

The error:

The timeout period elapsed prior to completion of the operation or the server is not responding.

The code:

string spath = @"c:\documents";
string[] files = Directory.GetFiles(spath, "*.*", SearchOption.AllDirectories);
Console.Write("Files Count:" + files.Length);

using (SqlConnection con = new SqlConnection(connectionString))
{
  con.Open();
  string insertSQL = "INSERT INTO table_Temp(doc_content, doc_path) values(@File, @path)";

  SqlCommand cmd = new SqlCommand(insertSQL, con);
  var pFile = cmd.Parameters.Add("@File", SqlDbType.VarBinary, -1);
  var pPath = cmd.Parameters.Add("@path", SqlDbType.Text);

  var tran = con.BeginTransaction();
  var fn = 0;

  foreach (string docPath in files)
  {
    string newPath = docPath.Remove(0, spath.Length);

    string archive = new DirectoryInfo(docPath).Parent.Name;
    fn += 1;

    using (var stream = new FileStream(docPath, FileMode.Open, FileAccess.Read))
    {
      pFile.Value = stream;
      pPath.Value = newPath;

      cmd.Transaction = tran;
      cmd.ExecuteNonQuery();

      if (fn % 10 == 0)
      {
        tran.Commit();
        tran = con.BeginTransaction();
        Console.Write("|");
      }

      Console.Write(".");
    }
  }

  tran.Commit();
}

Answer 1

For this, I would suggest using SQLBulkCopy methods, since it should be able to handle data insertion much more easily. Further, as others have pointed out, you might want to increase the timeout condition for your command.

Answer 2

While I would agree this may be best for a bulk copy of some sort, if you must do this in the c# program, your only option is probably to increase the timeout value. You can do this after your SqlCommand object has been created via cmd.CommandTimeout = <new timeout>; The property CommandTimeout is an integer representing the number seconds for the timeout, or zero if you never want it to timeout.

See the MSDN docs for details

Answer 3

You should be able to set the timeout for the transaction directly on that object in your application code, this way you are not changing sql server settings.

Secondly, you can build batches in your application also. You say you can get 80k docs before a timeout, set your batch size at 50k, process them, commit them, grab the next batch. Having your application manage batching also allows you to catch sql errors, such as timeout, and then dynamically adjust the batch size and retry without ever crashing. This is the entire reason for writing your application in the first place, other wise you could just use the wizard in management studio and manually insert your files.

I highly recommend batching over other options.

@Shubham Pandey also provides a great link to SQL bulk copy info, which also has links to more info. You should definitely experiment with the bulk copy class and see if you can get additional gains with it as well.