ADO Large DataSet throws out of memory exception

Question

I have a huge dataset (about 1.5 million records) that throwing a "out of memory exception", the dataset appears to load from the database rather quickly (2-3 seconds) but when the Merge() method is called the application crashes and throws a "Out Of Memory" exception.

DataSet dsTemp = setupBpc.GetAllData(); << THIS WORKS
if (dsTemp != null)
{
   dsSetupData.Merge(dsTemp); << OOM Exception
}

The application is old, I am somewhat stuck with ADO as refactoring would be a massive undertaking. It's a winforms application using Janus GridEX and I've not been able to find any sort of way to do paging with a GridEX component.

But the thing that stumps me is this doesn't appear to be an issue with the dataset itself, it's retrieving from the database and loading without issue. There seems to be something with the Merge() method that is causing this to fail.

Answer 1

You might run out of memory since you are merging data and even more memory is required. Think about it: First you hold the original DataSet, then you merge, which is basically copying the original dataset into the new one. Thus taking twice as much memory.

I am sure Janus GridEX supports paging parametering for data retrieval, and you should implement this through the whole data layer. I think this code change is unavoidable since this doesn't scale, and you have been lucky to not have this issue earlier!