How to read large excel files containing more that 200,000 rows and load that data in datatable in C#

Question

I am working on reading data from excel and loading it in datatable. My problem is that it is giving SystemOutOfMemoryException while loading large excel files. The colomns in excel are not fixed so I can't load that data in sql table. I need to do some manipulation on data so I am loading it in datatable. Can anyone suggest me how to resolve this issue?

I am doing it like this

 OleDbConnection conn = new OleDbConnection();
                OleDbCommand cmd = new OleDbCommand();
                OleDbDataAdapter da = new OleDbDataAdapter();
                conn = new OleDbConnection(GetOleDbConnectionString(strFileType, strNewPath));
                if (conn.State == ConnectionState.Closed) conn.Open();    
                string query = null;
                DataTable dt = new DataTable();              

                query = "SELECT  * FROM [" + SpreadSheetName + "]";               

                cmd.Connection = conn;
                cmd.CommandText = query;
                da.SelectCommand = cmd;
                da.Fill(dt);
                da.Dispose();
                conn.Close();
                conn.Dispose();

Answer 1

Your problem is not enough memory - likely your application runs as 32 bit app and all the stuff you load just is overloading it.

Make it a 64 bit application (in settings under the executable project) - and make sure you have physical memory adequate for a modern machine (8+gb).

Answer 2

You are loading the data of whole excel sheet in memory, never do that, few environment variables come into scenario like available memory that may change if application is deployed in another machine, and columns filled in excel sheet - what if all of the columns are used with lengthy text then few thousand records could be enough to run out of memory.

Better way is to fire a query to get only column name like

query = "SELECT  * FROM [" + SpreadSheetName + "]" where 1=2"

This will give you all the column names, use this to create a table in database. Once table is created either load records few at a time and do manipulation here on limited records and repeat till end.