concat two byte []返回System.OutOfMemoryException

Question

I have a problem with concat two byte[]. One of them have more than 300,000,000 byte. It's throwing exception of type System.OutOfMemoryException .

I use this code :

byte[] b3 = by2.Concat(by1).ToArray();

anybody can help me

Answer 1

Because of Concat call ToArray know nothing about how big the result array has to be. It can't create proper, big array and just fill it with data. So it creates small one, then when it's full creates new one with twice the size, etc. over and over again as long as there is more data to fill. This way you need much more memory then just theoretical (b1.Length + b2.Length) * 2 . And things get even more tricky, because after certain point these big arrays are allocated on LOH, and are not collected that easily by GC as normal objects.

That's why you should not use ToArray() in this case and do it the old-fashioned way: allocate new array with size equals combines sizes of source arrays and copy the data.

Something like:

var b3 = new byte[b1.Length + b2.Length];
Array.Copy(b1, b2, b1.Length);
Array.Copy(b1, 0, b2, b1.Length, b2.Length);

It does not guaranty success, but makes it more likely. And executes much, much, much faster then ToArray() .

Answer 2

When working with that amount of data, I think you should be working with streams (this of course depends on the application).

Then you can have code that works on the data without requiring it all to be loaded in memory at the same time, and you could create a specialized stream class that acts as a concatenation between two streams.

Answer 3

Well, the error message taks for itself, you don't have free continuous ~550Mb of RAM. Maybe it's just too fragmented.

Answer 4

Well.. you know, requesting from the system a continuous block of ~600meg - I'm not suprised. It is quite a large block itself, and provided that you must also have the source arrays in the memory, that's over 1GB of raw data chunks..

You should probably start thinking about other data structures, or try to keep them as files and map them to memory edit: memmapping a whole file needs the same contiguous area in address space, so it solves nothing. This answer will be deleted.