How can I export very large amount of data to excel

Question

I'm currently using EPPlus to export data to excel. It works admirably for small amount of data. But it consume a lots of memory for large amount of data to export.

I've briefly take a look at OOXML and/or the Microsoft Open XML SDK 2.5. I'm not sure I can use it to export data to Excel?

There is also third party provider libraries.

I wonder what solution could do the job properly of exporting very large amount of data in good performance and not taking to much spaces (ideally less than 3x the amount of data to export) ?

Update: some extra requirements... I need to be able to export "color" information (that exclude CSV) and I would like something easy to manage like EPPlus library (exclude the XML format itself). I found another thread and they recommend Aspose or SpreadsheetGear which I'm trying. I put first answer as ok. Thanks to all.

Update 2016-02-16 Just as information... We now use SpreadSheetGear and we love it. We required support once and it was awesome.

Thanks

Answer 1

EPPlus to export data to excel. It works admirably for small amount of data. But it consume a lots of memory for large amount of data to export.

A few years ago, I wrote a C# library to export data to Excel using the OpenXML library, and I faced the same situation.

It worked fine until you started to have about 30k+ rows, at which point, the libraries would be trying to cache all of your data... and it'd run out of memory.

However, I fixed the problem by using the OpenXmlWriter class. This writes the data directly into the Excel file (without caching it first) and is much more memory efficient.

And, as you'll see, the library is incredibly easy to use, just call one CreateExcelDocument function, and pass it a DataSet , DataTable or List<> :

// Step 1: Create a DataSet, and put some sample data in it
DataSet ds = CreateSampleData();

// Step 2: Create the Excel .xlsx file
try
{
    string excelFilename = "C:\\Sample.xlsx";
    CreateExcelFile.CreateExcelDocument(ds, excelFilename);
}
catch (Exception ex)
{ 
    MessageBox.Show("Couldn't create Excel file.\r\nException: " + ex.Message);
    return;
}

You can download the full source code for C# and VB.Net from here:

Mike's Export to Excel

Good luck !

Answer 2

If your requirements are simple enough, you can just use CSV.

If you need more detail, look into SpreadsheetML . It's an XML schema that you can use to create a text document that Excel can open natively. It supports formulas, multiple worksheets per workbook, formatting, etc.

Answer 3

I second using CSV but note that Excel has limits to the number of rows and columns in a worksheet as described here: http://office.microsoft.com/en-us/excel-help/excel-specifications-and-limits-HP010342495.aspx

specifically: Worksheet size 1,048,576 rows by 16,384 columns

This is for Excel 2010. Keep these limits in mind when working with very large amounts of data.

Answer 4

As an alternative you can use my SwiftExcel library. It was design for high volume Excel output that writes data directly to the file with no memory impact.

Here is a sample of usage:

using (var ew = new ExcelWriter("C:\\temp\\test.xlsx"))
{
    for (var row = 1; row <= 100; row++)
    {
        for (var col = 1; col <= 10; col++)
        {
            ew.Write($"row:{row}-col:{col}", col, row);
        }
    }
}