在 C# 中处理大列表时出现 OutOfMemoryException

Question

I have to process some big tabular data files (ex.: 300.000 rows by 200 columns), creating a data structure for each cell.

However, in a few thousands iterations a System.OutOfMemoryException is thrown (computer has 8GB RAM).

In my case, I parse each line as a List of Dictionary, since I must have the column name and the value, and process it by creating a node with these information.

Here is the code:

// foreach record 
foreach (Dictionary<string, string> inst in c.TodasInstancias)
{
instance = null;
tmp = null;

// for each column
foreach (KeyValuePair<string, string> kv in inst)
{
    if ((tmp = this.Relations.Find(new Predicate<Relacionamento>(x => x.Target.Name == kv.Key))) != null)
   { // if this class has a relationship with other classes
       p = og.CreateUriNode(this.UriOntologiaPrefix + ":" + kv.Key);
       o = og.CreateUriNode(new Uri(tmp.CampoOrigem.Classe.Configs.Name_space + "/" + kv.Value));
       og.Assert(instancia, p, o);
   }
   else (kv.Key != c.Configs.Identificador)
   { // se for um campo comum de dados
       p = og.CreateUriNode(this.UriOntologiaPrefix + ":" + kv.Key);
       o = og.CreateLiteralNode(kv.Value);
       og.Assert(instancia, p, o);
   }
}
}

Any hint on how to bypass this exception?

Answer 1

Use queryable to handle big lists. You can retrieve batch of records, instead of all data you need using Skip and Take. This way you will handle a small list in memory each time. Ex:

   var count=context.entities.Count(); 

   for(i=0,i<count,i+=1000)//1000 can be any size of batch
   {
       var batch =context.entities.Skip(i).Take(1000);
       //Do Operations you need
   }