如何管理大型h5文件

Question

I have been encountering this issue for a while. I have a h5 file with a dataset "ds", which consists of a matrix of size 170000x70000, float 16. Evidently, I cannot load all of this at once, but I do not need to. I only need to work with vectors of 170000x1 and filter out the elements that fulfill a condition. However, I am finding this to be very inefficient. I have tried to load it in chunks, like the example below, but this operation takes several minutes.

f=h5py.File(myfile, "r")
ds = f["ds"]
i=1000
for i in tqdm(range(0,170000,100)):
    chunk=ds[i:i+100,j]
    filtered_chunk =np.where(chunk>0)

Has any of you encountered this issue before? How can I tackle this?

Answer 1

This is how you read vectors of 170000x1. I think addresses your question

f=h5py.File(myfile, "r")
ds = f["ds"]
for cnt in range(ds.shape[1]):
    # get arr as a numpy array of shape (170000,1)
    arr=ds[:,cnt]  
    ...do something with arr