C# system.outofmemoryexception

Question

I have to deal with a table that has two columns and 80000 rows.I need to have a comparison on that table between the two columns which is resulting in a system.out of memory exception when I run it.

I will compare the first record value with rest of the 79,999 records ie(1,1),(1,2), .....(1,79999), (2,1),(2,2),....(2,79999)......(3,1),(3,2),....(3,79999)...

how to handle this situation

thanks in advance

Here is my code:

SqlCommand cmd = new SqlCommand("select (g.gene),n.goterm,(n.gene) from genematrix g  join genematrix n on n.goterm=g.goterm where g.id<n.id", con);
SqlDataAdapter da = new SqlDataAdapter(cmd);
DataSet ds = new DataSet();
da.Fill(ds);

Table name GeneMatrix:

GoTerm    Gene

1a       gene1
2b       gene1
1a       gene2
2b       gene3

so on till 80000 records...........

The above one is the table structure from which i will be comparing first Gene record with all other gene records and then go for the second gene record and start comparing it to all other records...so on till the last gene.

when i compare gene1 with gene2 for common go terms the result is like

gene1 gene2 1a
gene1 gene3 2b

My above query returns the output correctly ,but the problem is ..a system.outofmemoryexception is shown and i couldn't get the query run completely till the end.

Answer 1

Check that you are building a 64-bit process, and not a 32-bit one, which is the default compilation mode of Visual Studio. To do this, right click on your project, Properties -> Build -> platform target : x64. As any 32-bit process, Visual Studio applications compiled in 32-bit have a virtual memory limit of 2GB.

64-bit processes do not have this limitation, as they use 64-bit pointers, so their theoretical maximum address space is 16 exabytes (2^64). In reality, Windows x64 limits the virtual memory of processes to 8TB. The solution to the memory limit problem is then to compile in 64-bit.

However, object's size in Visual Studio is still limited to 2GB, by default. You will be able to create several arrays whose combined size will be greater than 2GB, but you cannot by default create arrays bigger than 2GB. Hopefully, if you still want to create arrays bigger than 2GB, you can do it by adding the following code to you app.config file:

<configuration>
  <runtime>
    <gcAllowVeryLargeObjects enabled="true" />
  </runtime>
</configuration>

Answer 2

I don't think you describing you issue correctly. You start with a table with 2 columns, but your snippet loads a join with 3 columns.

Please show the line which results in the OutOfMemory exception.

Theory 1. Run a count query: select count(g.gene)from genematrix g join genematrix n on n.goterm=g.goterm where g.id

Theory 2. If loading the records is the problem, then I think you building 80000*80000 array in your code. Even if your entry was only one byte, 80000*80000b = 6400000000b = 5.9GB which more the .Net 'single object 2GB' limit.

Answer 3

You should split up the comparison code into smaller parts where you fetch eg 1000 records at a time, do the comparison on these and then get the next 1000 records.

The problem you're encountering is most likely due to the large amounts of records you're fetching at once.

Answer 4

You did not specify what you are comparing for, or what you should do if you find a match.

In any case i would suggest you use the Parallel LINQ extensions . They seriously kick ass and make mincemeat out of a workload like this, and it has a lot of smarts built in to pick the best approach for the work.

Answer 5

Use the DataReader instead of Dataset it will increase you application performance and reduce system overheads. This is due to one row at a time is stored in memory. You create a DataReader by calling Command.ExecuteReader after creating an instance of the Command object. Look at http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldatareader.aspx

Answer 6

You should not use a cross join. Better load your table once into main memory like this:

SqlCommand cmd 
= new SqlCommand("select g.goterm,g.gene from genematrix g ", con);
    SqlDataAdapter da = new SqlDataAdapter(cmd);
    DataSet ds = new DataSet();
    da.Fill(ds);

And then make the comparison by using a Dictionary. Here is some "air code" (untested, uncompiled, just out of my head):

var d = new Dictionary<string, List<string>();
for each(var row in ds.Rows)
{
    string key = (string)row["goterm"];
    if(!d.ContainsKey(key))
        d.Add(key, new List<string>());
     d[key].Add((string)row["gene"]);
}

for each(var k in d.Keys)
{
     Console.Write(k + ": ");
     for each(var gene in d[k])
          Console.Write(gene + " ");
     Console.WriteLine();
}

(ALAS, if this is not just working out-of-the box, it should give you a rough idea how to solve it).