CodeGo.net>如何提高foreach循环的性能

Question

I have this c# program where I am able to export data to an excel file. It calls the stored procedure in sql server and returns the results to the excel file (based on the filter I choose).

Right now, I am able to export it successfully but if I try to extract way more data ( 20,000 records or so) from a 20 million record table, it is taking way way too long. I added a stopwatch to the code and found that the foreach loop is the culprit here.

Here is my code:

private void ExportExcel(SqlDataReader dr)
    {
        try
        {
            DataTable dt = new DataTable();
            dt.Load(dr);
            SaveFileDialog saveFileDialog1 = new SaveFileDialog();
            saveFileDialog1.Filter = "Microsoft Office Excel Workbook (*.xls)|*.xls|All Files (*.*)|*.*";
            saveFileDialog1.FilterIndex = 1;
            saveFileDialog1.RestoreDirectory = true;

            if (saveFileDialog1.ShowDialog() == DialogResult.OK)
            {
                // Create an Excel object and add workbook...
                Excel.ApplicationClass excel = new Excel.ApplicationClass();
                Excel.Workbook workbook = excel.Application.Workbooks.Add(true); // true for object template???


               // var watch = System.Diagnostics.Stopwatch.StartNew();
                // Add column headings...
                int iCol = 0;
                int iVisibleColumnCount = 0;
                foreach (DataColumn c in dt.Columns)
                {
                    iCol++;
                    // counting visible columns
                    if (c.ColumnMapping != MappingType.Hidden)
                        iVisibleColumnCount++;
                    else    // hide the columns in excel if the column is hide in datatable
                    {
                        ((Excel.Range)excel.Cells[1, iCol]).EntireColumn.Hidden = true;
                        continue;
                    }
                    // Set column header text to bold
                    ((Excel.Range)excel.Cells[1, iCol]).Font.Bold = true;
                    excel.Cells[1, iCol] = c.ColumnName;

                    if (c.DataType == typeof(System.String))
                        ((Excel.Range)excel.Cells[1, iCol]).EntireColumn.NumberFormat = "@";
                    else if (c.DataType == typeof(System.Int16)
                        || c.DataType == typeof(System.Int32)
                        || c.DataType == typeof(System.Int64))
                        ((Excel.Range)excel.Cells[1, iCol]).EntireColumn.NumberFormat = "#,##0";
                    else if (c.DataType == typeof(System.TimeSpan))
                        ((Excel.Range)excel.Cells[1, iCol]).EntireColumn.NumberFormat = @"[$-409]hh:mm:ss AM/PM;@";
                    else if (c.DataType == typeof(System.DateTime))
                        ((Excel.Range)excel.Cells[1, iCol]).EntireColumn.NumberFormat = "yyyy-mm-dd hh:mm:ss";
                    else if (c.DataType == typeof(System.Decimal))
                        ((Excel.Range)excel.Cells[1, iCol]).EntireColumn.NumberFormat = @"#,##0.00_);[Red](#,##0.00)";
                    else
                        ((Excel.Range)excel.Cells[1, iCol]).EntireColumn.NumberFormat = "General";
                }
                // for each row of data...
                int iRow = 0;
                foreach (DataRow r in dt.Rows)
                {
                    iRow++;

                    // add each row's cell data...
                    iCol = 0;
                    foreach (DataColumn c in dt.Columns)
                    {
                        iCol++;
                        if (c.ColumnMapping != MappingType.Hidden)
                        {
                            if (c.DataType == typeof(DateTime))
                            {
                                DateTime date1 = (DateTime)r[c.ColumnName];
                                string DateTime = date1.ToString();

                                if (DateTime.Contains("AM"))
                                {
                                    excel.Cells[iRow + 1, iCol] = date1.ToString("yyyy-MM-dd hh:mm:ss") + " AM";
                                }
                                else   {
                                    excel.Cells[iRow + 1, iCol] = date1.ToString("yyyy-MM-dd hh:mm:ss") + " PM";
                                }


                            }
                            else {
                                excel.Cells[iRow + 1, iCol] = r[c.ColumnName].ToString();
                            }
                        }
                    }
                }

Can anyone give some suggestion on how to improve the performance it takes to export to excel?

Answer 1

How are you accessing Excel - via COM interop?

If so, you'll want to avoid setting each cell's value individually, because that is very slow indeed. Try to assign arrays to ranges as large as you possibly can.

For example, since you seem to iterate over each row unconditionally, it looks like you could at least try build an array/vector representing one column, and assigning it to an appropriate range in that column.

Answer 2

There's no obvious way to improve this. If the loop is taking the time (rather than the sql procedure) then the majority of the work must be being done by the excel library. You could look into different libraries for excel, and if you're lucky one might allow building rows in parallel, or just be faster. Otherwise you're limited to making small optimisations on your own code inside the loop. You could probably improve that date time code a bit, but I'd be surprised if it would make much difference.

Do you need an excel sheet or would a csv (which can be opened by excel) do? If a csv would work then it might be possible to export from sql directly or build the rows in parallel yourself without relying on libraries.