Dask：读取 hdf5 并写入其他 hdf5 文件

Question

I am working with a hdf5 file that is larger than memory. Therefore, I'm trying to use dask to modify it. My goal is to load the file, do some modifications (not necessarily preserving shape), and saving it to some other file. I create my file with:

import h5py as h5
import numpy as np

source_file = "source.hdf5"
x = np.zeros((3, 3))  # In practice, x will be larger than memory
with h5.File(source_file, "w") as f:
    f.create_dataset("/x", data=x, compression="gzip")

Then, I use the following code to load, modify and save it.

from dask import array as da
import h5py as h5
from dask.distributed import Client


if __name__ == "__main__":
    dask_client = Client(n_workers=1)  # No need to parallelize, just interested in dask for memory-purposes

    source_file = "source.hdf5"
    temp_filename = "target.hdf5"

    # Load dataframe
    f = h5.File(source_file, "r")
    x_da = da.from_array(f["/x"])

    # Do some modifications
    x_da = x_da * 2

    # Save to target
    x_da.to_hdf5(temp_filename, "/x", compression="gzip")

    # Close original file
    f.close()

However, this gives the following error:

TypeError: ('Could not serialize object of type Dataset.', '<HDF5 dataset "x": shape (3, 3), type "<f8">') distributed.comm.utils - ERROR - ('Could not serialize object of type Dataset.', '<HDF5 dataset "x": shape (3, 3), type "<f8">')

Am I doing something wrong, or is this simply not possible? And if so, is there some workaround?

Thanks in advance!

Answer 1

For anyone interested, I created a workaround which simply calls compute() on each block. Just sharing it, although I'm still interested in a better solution.

def to_hdf5(x, filename, datapath):
    """
    Appends dask array to hdf5 file
    """
    with h5.File(filename, "a") as f:
        dset = f.require_dataset(datapath, shape=x.shape, dtype=x.dtype)

        for block_ids in product(*[range(num) for num in x.numblocks]):
            pos = [sum(x.chunks[dim][0 : block_ids[dim]]) for dim in range(len(block_ids))]
            block = x.blocks[block_ids]
            slices = tuple(slice(pos[i], pos[i] + block.shape[i]) for i in range(len(block_ids)))
            dset[slices] = block.compute()