OutOfMemoryException - Dataset <-> XML <-> Dataset

Question

I'm stuck in a huge problem where i need to handle huge data.

I get a data set which has 5 datatables. One of the tables has around 3000 columns and 50,000 records.

I need to save this dataset completely in the SQL database as XML and i need to retrieve it later.

I CAN'T make any design changes to skip it. When i do dataset.GetXml() it returns a string of huge length which throws OutOfMemoryException. I knew that the string datatype has a certain limit to carry data.

But in sql table the xml column can hold upto 2 Gb. So how can i move this dataset back and forth between the database and my application?

Thank you

NLV

Answer 1

Your design sounds flawed. Why do you need to retrieve the entire table into memory? Page through it instead - operate on the table in smaller batches - say, 1000 records at a time.

You probably don't even need to do that. Generally, database applications operate over just the data that's changing. If your app needs to modify 10 rows, then retrieve just those ten rows. You'll save not only memory, but all the time it takes to retrieve and post that data.

3000 columns in the table? That also smells (really) bad. Look into normalizing your database.

I know it feels easier to fix the immediate problem and move on. You'll have many, many fewer problems in the long run, however, if you fix the (frankly quite severe) design problems up front.

Answer 2

You will need to switch to all the APIs that involve streaming, use SqlDataReader (1 row at a time), not DataSet. Also, when possible, stream you data to temporary files on the file system instead of holding it in memory.

If 1 row is too much to hold, then likely you need to use the streaming APIs to talk to the blobs in the relevant large columns.

Answer 3

3k columns x 50k rows = 150 million elements before you start. How many characters per element?

I have to ask... why are you using a database to store this? You may as well store it on a fileserver.

50k rows is tiny, but 3k columns is extreme, and it's all hugely bloated in XML