如何在C ++中将稀疏数组有效地保存在文件中？

Question

I have an array of doubles having 6 indices, and it is mostly filled with zeros. I don't know yet what type should I use to storage it in the memory.

But, most importantly: I would like to save it into a file (a binary file?). What is the most efficient way to save it? One requirement is that I can run through all the non-zero entries without passing by the zeros. If I run 6 nested for I'll need too many lives.

Moreover, I don't know how to practically save it: Do I need two files, one acting as an index and the second one containing all the values?

Thanks!

Answer 1

This is probably a solved problem; there are probably sparse-matrix libraries that give you efficient in-memory representations too. (eg each row is a list of index:value , stored in a std::vector , linked list, hash, or other data structure, depending on whether inserting single non-zero values in the middle is valuable or whatever other operation is important).

---

A binary format will be faster to store/load, but whether you go binary or text isn't important for some ways of representing a sparse array. If you write a binary format, endian-agnostic code is a good way to make sure it's portable and doesn't have bugs that only show up on some architectures.

Options:

• Simple but kind of ugly: gzip / lz4 / lzma the buffer holding your multidimensional array, writing the result to disk. Convert to little-endian on the fly while saving/loading, or store an endianness flag in the format.

• Same idea but store all 6 indices with each value. Good if many inner-most arrays have no non-zero values, this may be good. Every non-zero value has a separate record (line, in a text-based format). Sample line (triple-nested example for readability, extends to 6 just fine):

dimensions on the first line or something
a b c  val
...
3 2 5   -3.1416

means: matrix[3][2][5] = -3.1416

• Use a nested sparse-array representation: each row is a list of index:value. Non-present indices are zero. A text format could use spaces and newlines to separate things; a binary format could use a length field at the start of each row or a sentinel value at the end.

  You could flatten the multidimensional array out to one linear index for storage with 32bit integer indices, or you could represent the nesting somehow. I'm not going to try to make up a text format for this, since it got ugly as I started to think about it.

Answer 2

A regular flat representation of a 6 dimension array ...

double[10][10][10][10][10][10] = 1million entries * 8 bytes ~= 8MB

An associative array Index:Value representation, assume 50% of entries are 0.0 ... using a 4 byte 32bit index ...

500,000 * 4 bytes + 500,000 * bytes ~= 6MB

A bit map representation of the sparse array, assume 50% of entries are 0.0 ... bits are set so that every byte represents 8 entries in the array 10000001b would mean 8 entries where only the first and last are represented and the 6 middle values are ignored since they are zero ...

ceil(1million / 8) bytes + 500,000 * 8 bytes ~= 4.125MB