How to deal with large arrays in python?

Question

I have very large arrays which are needed to be stored on disk. How can I do that? I tried using h5py, I read the documentation and found the issue with it is its 2nd dimension can have maximum number of 1024. How can I deal with an array that have dimensions of lets say 10^5 x 10^5?

Answer 1

Assuming you read this part of the documentation . This does not say that there is a general maximum size of 1024 for the second dimension. It's just an example of using resizable datasets.

The per-axis limit for HDF5 (and h5py) is 2^64 elements, as can be read in the documentationon on resizable datasets .