在C＃中将大型双精度数组保存为文件的最有效方法

Question

I'm writing a program that is reading data from .dat files into double[,,] arrays, calculates some stuff and needs to write the arrays into a file to save them for a later usage.

These arrays can have up to [64x64x150000] elements which forces me to already load those files in small parts into the program to make use of them (otherwise the MemoryException is called). Until now I used textfiles to save smaller arrays on my harddisk but saving a [64x64x150000] array step by step fills up above >6GB per file at the end which is quiet a lot when you have to work with a lot of those .dat-files and have pretty much to keep all the .txt-files.

So I would like to know if any other filetype saves some harddisk space or if there is another possibility to save those arrays outside of my program for a later usage with less harddisk space requirement.

(I need to be able to exchange the files between different computers).

Answer 1

(8 B/double * (64 * 64 * 150000) double) / (10 ⁹ B/GB) = 5.6 GB

So unless you either reduce to a lower precision (floats) or perform some kind of compression, you'll need 5.6 GB to store all those doubles. Reducing to floats would take 2.8 GB per file.

For each of the 64 * 64 vectors of length 150000, you may be able to perform a signal compression (depending on what the data looks like). That's a broad topic, so without knowing more all I can give you is a starting point: Signal compression .

Answer 2

Either compression, or try Binary Serialization . A double can take up dozens of bytes in text, particularly depending on your encoding (1-2 per digit). In binary, each one is exactly 8 bytes (+ however much overhead for bookkeeping, probably minimal).