SqlBulkCopy Azure DataTable faster than Streaming

Question

I need a worker role that takes a txt file from a Blob and bulk it into an Azure Sql table.

I'm using the SqlBulkCopy provided by LumenWorks.Framework.IO, I've created 2 version of the worker role: 1) read the entire file, load it into a DataTable, execute the SqlBulkCopy 2) StreamRead the file and pass the Stream to the SqlBulkCopy

The problem is that the second version has like half of the performance of the first one.

As an example with a 10MB txt file, with 90'000 records: -first version: half a second to load file, 2 seconds to convert to a DataTable, 20 seconds for the SqlBulkCopy

-second version: 50 seconds total (more than double!)

I've tried to change the BatchSize but it doesn't seem to make much difference and I don't know what im doing wrong, here is the 2' version code:

using (var stream = await blockBlob.OpenReadAsync(cancellationToken))
using (var reader = new StreamReader(stream))
using (var csv = new CsvReader(reader, true, ';'))
using (var conn = new SqlConnection(CloudConfigurationManager.GetSetting("TestDbConn")))
{
      await conn.OpenAsync(cancellationToken);
      connAperta = true;
      using (var transaction = conn.BeginTransaction())
      using (var bulkCopy = new SqlBulkCopy(conn, SqlBulkCopyOptions.KeepIdentity | SqlBulkCopyOptions.TableLock, transaction))
      {
             bulkCopy.BulkCopyTimeout = 300;
             bulkCopy.DestinationTableName = "[3sc_base_1]";
             await bulkCopy.WriteToServerAsync(csv, cancellationToken);
             transaction.Commit();
      }
}

What i'm doing wrong??

Answer 1

Have a look at the new Azure SQL Database capability to bulk upload directly from an Azure Storage account .

This should be the fastest and easiest way to achieve what you want unless you are not streaming directly but doing transformation as well.