Copying large datatable into excel in ASP.Net

Question

I've written some code that copies the data from an Azure database into an Excel file. This can be found at the end of this question.

The problem is it takes forever to populate an excel sheet when I have 10k rows for one of the tables. Obviously, this is not ideal for Excel but at this point it has to be done this way. I'm wondering if there is a faster way to code this. Certainly, creating excel sheet is the bottleneck, because C# grabs the dataset in seconds. If I go into Excel and view the data and then right click and copy with headers and paste that into and excel sheet it also does this in seconds.

So can I programmatically do this?

private void createExcelFile()
        {
            string fileName = "FvGReport.xlsx";
            string filePath = HttpContext.Current.Request.MapPath("~/App_Data/" + fileName); //check www.dotnetperls.com/mappath
            string sqlQuery = "";
            List<string> sheetNames = new List<string>();

            foreach (ListItem item in ddlSummary_Supplier.Items)
            {
                string sqlSummary = "SELECT * FROM FvGSummaryAll WHERE Supplier_Code = '" + item.Text + "'; ";
                sqlQuery = sqlQuery + sqlSummary;
                sheetNames.Add("Summary " + item.Text);

                string sqlPaymentsSummary = "SELECT * FROM FvGSummaryPayment WHERE Supplier_Code = '" + item.Text + "'; ";
                sqlQuery = sqlQuery + sqlPaymentsSummary;
                sheetNames.Add("PaymentSummary " + item.Text);
            }

            DataSet dataSet = new DataSet();
            //string sqlQuery = @"SELECT * FROM FvGData WHERE Supplier_Code = 'SFF Pacific'; SELECT * FROM FvGSummaryPayment";


            using (SqlConnection connection = new SqlConnection(connectionString))
            {
                connection.Open();
                SqlDataAdapter adapter = new SqlDataAdapter();
                adapter.SelectCommand = new SqlCommand(sqlQuery, connection);
                adapter.Fill(dataSet);
            }

            //this reference conflicts with System.Data as both have DataTable. So defining it here. 
            Microsoft.Office.Interop.Excel.Application ExcelApp = new Microsoft.Office.Interop.Excel.Application();
            Microsoft.Office.Interop.Excel.Workbook excelWorkBook = null;
            Microsoft.Office.Interop.Excel.Worksheet excelWorkSheet = null;
            ExcelApp.Visible = true;
            excelWorkBook = ExcelApp.Workbooks.Add(Microsoft.Office.Interop.Excel.XlWBATemplate.xlWBATWorksheet);


            //excel rows start at 1 not 0
            try
            {
                for (int i = 1; i < dataSet.Tables.Count; i++)
                {
                    excelWorkBook.Worksheets.Add();  //Adds new sheet in Excel WorkBook
                }

                for (int i = 0; i < dataSet.Tables.Count; i++)
                {
                    int dsRow = 1;
                    excelWorkSheet = excelWorkBook.Worksheets[i + 1];

                    //Writing Columns Name in Excel Sheet
                    for (int col = 1; col < dataSet.Tables[i].Columns.Count; col++)
                    {
                        excelWorkSheet.Cells[dsRow, col] = dataSet.Tables[i].Columns[col - 1].ColumnName;
                    }
                    dsRow++;

                    for (int xlRow = 0; xlRow < dataSet.Tables[i].Rows.Count; xlRow++)
                    {
                        //Excel row and col positions for writing row = 1, col = 1
                        for (int col = 1; col < dataSet.Tables[i].Columns.Count; col++)
                        {
                            excelWorkSheet.Cells[dsRow, col] = dataSet.Tables[i].Rows[xlRow][col - 1].ToString();                           
                        }
                        dsRow++;
                    }

                    excelWorkSheet.Name = sheetNames[i]; //Renaming ExcelSheets
                }

                excelWorkBook.SaveAs(filePath);
                excelWorkBook.Close();
                ExcelApp.Quit();
                Marshal.ReleaseComObject(excelWorkSheet);
                Marshal.ReleaseComObject(excelWorkBook);
                Marshal.ReleaseComObject(ExcelApp);
            }
            catch (Exception ex)
            {
                lblNoData.Text = ex.ToString();
            }
            finally
            {
                foreach (Process process in Process.GetProcessesByName("Excel"))
                {
                    process.Kill();
                }
            }

            downloadExcel(filePath, fileName);
        }

Answer 1

It looks like you're using Office Automation, which is usually slow on things like this, in my experience. I would suggest saving the output as a delimited file (.csv) and using automation to open that file (or files) with Excel and then save it as a spreadsheet.

Answer 2

I would suggest you to try using some ETL tool, Esspecially if you will do this from time to time again. If you take Talend, for instance, .... you connect to the DB and the schema will be pulled on its own. Take a SQL input component and connect it to a Excel component and you are done. Take about 5 minutes, without a single line of code

Answer 3

I'm not sure what you mean by 'forever', but for comparison I have a process that writes an OpenXML Spreadsheet of 46,124 row with ~500 characters per row in less than 17-seconds. This is generated on by a C# process that is on a separate server at the same hosting facility as the database server.

If writing to a CSV is an option, then that is going to be the solution with the best performance. OpenXML will give you the next best performance, I found the following article to be the most helpful when I was trying to put together my process:

Read-and-Write-Microsoft-Excel-with-Open-XML-SDK

Regarding memory -- You have two things you need to put in memory, your incoming data and your outgoing file. Regardless of what type of file you write, you'll want to use a SqlDataReader instead of your dataSet . This means your incoming data will only have one row at a time in memory (instead of all 10K). When writing your file (CSV or OpenXML) if you write directly to disk (FileStream) instead of memory (MemoryStream) you'll only have that little bit in memory.

Especially if you are running code that is running within your website, you don't want to use up a bunch of memory at once because .NET/IIS doesn't handle that very well.