从文本文件读取数据时抛出“System.OutOfMemoryException”类型的异常

Question

I m trying to read a text file data with "|"separated, I m using below code. I m able to read the data from text file after reading when trying to put datatable data into dataview I m getting Exception of type 'System.OutOfMemoryException' was thrown,

Can any one suggest me how to avoid this exception.

string filepath = System.Configuration.ConfigurationManager.AppSettings["data"];  

if (filepath != "")
            {
                DataTable dt = new DataTable("file");
                string[] columns1 = null;
                var lines = File.ReadAllLines(filepath);
                int Count = lines.Length;
                //here taking columns and adding to table
                 if (lines.Count() > 0)
                {
                    columns1 = lines[0].Split(new char[] { '|' });
                    foreach (var column in columns1)
                        dt.Columns.Add(column);
                }                   
                 for (int i = 1; i < lines.Count(); i++)
                {
                    DataRow dr = dt.NewRow();
                    string[] values = lines[i].Split(new char[] { '|'  });
                    for (int j = 0; j < values.Count() && j < columns1.Count(); j++)
                    {
                      dr[j] = values[j];
                    }

                }
                 dt.Rows.Add(dr);   

                    DataView View = new DataView(dt);
                    //Here I m getting "Exception of type 'System.OutOfMemoryException' was thrown."
                    DataTable MD = View.ToTable("MD", false, "ID", "Description")
                    DataTable MM = View.ToTable("MM", false, "RecordNumber", "Item description")
                if (MD.Rows.Count > 0)
                {
                    InsertData(MD);
                }
                if (MM.Rows.Count > 0)
                {
                    InsertData1(MM);
                }
            }

StackTrace :-

    at System.Collections.Generic.List`1.set_Capacity(Int32 value)
    at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
    at System.Collections.Generic.List`1.Add(T item)
    at System.Data.DataView.ToTable(String tableName, Boolean distinct, String[] columnNames)

Answer 1

Multiple uses of String.Split are likely to cause an OutOfMemoryException. Download Lumenworks Fast CSV Reader - problem solved. You will get it here: http://www.codeproject.com/Articles/9258/A-Fast-CSV-Reader

For more details on issues with Split, have a look at the following links:

http://vpnchoudhary.blogspot.ie/2011/03/out-of-memory-exception-simple.html

string.split() "Out of memory exception" when reading tab separated file

Relevant quote here:

What happens when you do a split on a string containing 1355049 comma separated strings of 16 characters each, having total character length of 25745930 ?

An Array of pointers to string object: Contiguous virtual address space of 4 (address pointer)*1355049 = 5420196 (arrays size) + 16 (for book keeping) = 5420212. Non-contiguous virtual address space for 1355049 strings, each of 54 bytes. It does not mean all those 1.3 million strings would be scattered all across the heap, but they will not be allocated on LOH. GC will allocate them on bunches on Gen0 heap. Split.Function will create internal array of System.Int32[] of size 25745930, consuming (102983736 bytes) ~98MB of LOH, which is very expensive L.