Code Optimization when importing Excel Spreadsheet with Epplus

Question

I have need to import upwards of 35,000 Codes for a campaign from an excel spreadsheet. I have the following code below (which works in exactly the way i need it to) but when i run the process it can take over 20-30 mins to complete.

If there is anything that can be done to help speed up the process that would be greatly appreciated. I wouldn't call myself a advanced programmer and i know this could probably be done with advance coding techniques. If someone could point me in the right direction that would be greatly appreciated.

The models for the campaign and mastercode tables.

public class Campaign
{
    public int CampaignId { get; set; }
    public string Name { get; set; }
    public virtual List<MasterCode> MasterCodes { get; set; }
}

public class MasterCode
{
    public int MasterCodeId { get; set; }

    public int CampaignId { get; set; }
    public virtual Campaign Campaign { get; set; }

    public string Code { get; set; }
    public bool Used { get; set; }
    public DateTime SubmittedOn { get; set; }
}

The following is the code in the view. The form fields are not model bound as this code is in a popup window on a view that is already model bound to another model.

 @using (Html.BeginForm("UploadMasterCodes", "Campaigns", FormMethod.Post, new { enctype = "multipart/form-data" }))
{
    @Html.AntiForgeryToken()
    @Html.Hidden("CampaignId", Model.CampaignId)

    <div class="form-group">
        @Html.Label("Master Code File")
        @Html.TextBox("File", null, new { type = "file", @class = "form-control" })
    </div>
}

Controller Code

[HttpPost]
[ValidateAntiForgeryToken]
public ActionResult UploadMasterCodes(CampaignMasterCodeUploadViewModel model)
{
    if (ModelState.IsValid)
    {
        var result = CampaignMethods.uploadMasterCodes(model.File, model.CampaignId);
        TempData["SuccessMessage"] = result;
        return RedirectToAction("Details", new { id = model.CampaignId });
    }

    return RedirectToAction("Details", new { id = model.CampaignId });
}

And finally here is the code that performs the upload to the database. I am building a string that will track any errors that happen as i need to know of them all.

    public static string uploadMasterCodes(HttpPostedFileBase file, int campaignId)
    {
        using (ApplicationDbContext _context = new ApplicationDbContext())
        {
            string response = string.Empty;

            var campaign = _context.Campaigns.Find(campaignId);
            if (campaign == null)
            {
                return String.Format("Campaign {0} not found", campaignId.ToString());
            }

            var submitDate = DateTime.Now;
            int successCount = 0;
            int errorCount = 0;

            if ((file != null) && (file.ContentLength > 0) && !string.IsNullOrEmpty(file.FileName))
            {
                byte[] fileBytes = new byte[file.ContentLength];
                var data = file.InputStream.Read(fileBytes, 0, Convert.ToInt32(file.ContentLength));

                using (var package = new ExcelPackage(file.InputStream))
                {
                    var currentSheet = package.Workbook.Worksheets;
                    var workSheet = currentSheet.First();
                    var noOfRow = workSheet.Dimension.End.Row;

                    for (int i = 1; i <= noOfRow; i++)
                    {
                        var masterCode = new MasterCode();
                        masterCode.Code = workSheet.Cells[i, 1].Value.ToString();
                        masterCode.CampaignId = campaignId;
                        masterCode.Campaign = campaign;
                        masterCode.SubmittedOn = submitDate;

                        // Check to make sure that the Code does not already exist.
                        if (!campaign.MasterCodes.Any(m => m.Code == masterCode.Code))
                        {
                            try
                            {
                                _context.MasterCodes.Add(masterCode);
                                _context.SaveChanges();
                                successCount++;
                            }
                            catch (Exception ex)
                            {
                                response += String.Format("Code: {0} failed with error: {1} <br />", masterCode.Code, ex.Message);
                                errorCount++;
                            }
                        }
                        else
                        {
                            response += String.Format("Code: {0} already exists <br />", masterCode.Code);
                            errorCount++;
                        }
                    }

                    response += string.Format("Number of codes:{0} / Success: {1} / Errors {2}", noOfRow.ToString(), successCount.ToString(), errorCount.ToString());
                }
            }

            return response;
        }
    }

Answer 1

Whenever I need to read an Excel file I use this OLEDB provider:

Provider=Microsoft.ACE.OLEDB.12.0;Data Source={FilePath};Extended Properties='Excel 12.0 Xml;HDR=Yes;IMEX=1';

(More information on this topic can be found here and here )

Then you could do a bulk insert for the quickest insert. See this answer how to do this in memory. In your case you'll first need to save the file somewhere on the server, process and report progress to the user.

Answer 2

On the suggestion of @grmbl, and lots of reading, I was able to solve the speed issue using SQLBulkCopy. I didn't use the OLEDB provider but instead copied the file to the server and created a datatable. Which i then used for the bulk copy. The following code got the runtime down to around 10 seconds for 335,000 records.

I have cut out all error checking code so its not a wall of code. The controller code.

    [HttpPost]
    [ValidateAntiForgeryToken]
    public ActionResult UploadMasterCodes(CampaignMasterCodeUploadViewModel model)
    {
        if (ModelState.IsValid)
        {
            var filename = Path.GetFileName(model.File.FileName);
            var path = FileMethods.UploadFile(model.File, Server.MapPath("~/App_Data/Bsa4/"), filename);
            var dt = CampaignMethods.ProcessMasterCodeCsvToDatatable(path, model.CampaignId);
            TempData["SuccessMessage"] = CampaignMethods.ProcessMastercodeSqlBulkCopy(dt);
            return RedirectToAction("Details", new { id = model.CampaignId });                
        }

        TempData["ErrorMessage"] = "Master code upload form error.  Please refresh the page and try again.";
        return RedirectToAction("Details", new { id = model.CampaignId });
    }

The processing code.

    public static DataTable ProcessMasterCodeCsvToDatatable(string file, int campaignId)
    {
        using (ApplicationDbContext _context = new ApplicationDbContext()) {

            var campaign = _context.Campaigns.Find(campaignId);

            DataTable dt = new DataTable();
            dt.Columns.Add("CampaignId");
            dt.Columns.Add("Code");
            dt.Columns.Add("Used");
            dt.Columns.Add("SubmittedOn");

            string line = null;
            var submitDate = DateTime.Now;

            using (StreamReader sr = File.OpenText(file))
            {
                while ((line = sr.ReadLine()) != null)
                {
                    string[] data = line.Split(',');
                    if (data.Length > 0)
                    {
                        if (!string.IsNullOrEmpty(data[0]))
                        {
                            DataRow row = dt.NewRow();
                            row[0] = campaign.CampaignId;
                            row[1] = data[0];
                            row[2] = false;
                            row[3] = submitDate;

                            dt.Rows.Add(row);
                        }
                    }
                }
            }

            return dt;
        }
    }

    public static String ProcessMastercodeSqlBulkCopy(DataTable dt)
    {
        string Feedback = string.Empty;

        using (SqlConnection cn = new SqlConnection(ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString))
        {
            cn.Open();
            using (SqlBulkCopy copy = new SqlBulkCopy(cn))
            {
                copy.ColumnMappings.Add(0, "CampaignId");
                copy.ColumnMappings.Add(2, "Code");
                copy.ColumnMappings.Add(3, "Used");
                copy.ColumnMappings.Add(4, "SubmittedOn");
                copy.DestinationTableName = "MasterCodes";
                try
                {
                    //Send it to the server
                    copy.WriteToServer(dt);
                    Feedback = "Master Code Upload completed successfully";
                }
                catch (Exception ex)
                {
                    Feedback = ex.Message;
                }
            }
        }

        return Feedback;
    }