如何使用 h5py 将值并行添加到现有的 HDF5 文件中，其中包含 3 个组和每个组中的 12 个数据集？

Question

I have installed the libraries using this link . I have already created an HDF5 file called test.h5 using mpiexec -n 1 python3 test.py . test.py is as below and I'm not sure if it is necessary to use mpi4py here, please let me know.

from mpi4py import MPI
import h5py

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

f = h5py.File('test.h5', 'w', driver='mpio', comm=comm)

f.create_group('t1')
f.create_group('t2')
f.create_group('t3')

for i in range(12):
    f['t1'].create_dataset('test{0}'.format(i), (1,), dtype='f', compression='gzip')
    f['t2'].create_dataset('test{0}'.format(i), (1,), dtype='i', compression='gzip')
    f['t3'].create_dataset('test{0}'.format(i), (1,), dtype='i', compression='gzip')

f.close()

Now, I would like to write a test1.py file that will:

1. Open test.h5 and get all the unique keys (they are the same for all three groups).
2. Make chunks of those keys, like chunks = [['test0','test1','test2'],['test3','test4','test5'],['test6','test7','test8'],['test9','test10','test11']] . I don't care about the the order or groupings of these chunks but I would like one chunk per process.
3. For each chunk assign a process to store a value for every key in that chunk in every group. In other words I would like to run this function in parallel:

def write_h5(f, rank, chunks):
    for key in chunks[rank]:
        f['t1'][key][:] += 0.5
        f['t2'][key][:] += 1
        f['t3'][key][:] += 1

How do I do this? Can you please explain in detail? Thanks a lot in advance!

Answer 1

test1.py should contain:

from mpi4py import MPI
import h5py

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

def chunk_seq(seq, num):
    avg = len(seq) / float(num)
    out = []
    last = 0.0
    while last < len(seq):
        out.append(seq[int(last):int(last + avg)])
        last += avg
    return out

def write_h5(f, chunk):
    for key in chunk:
        f['t1'][key][:] += 0.5
        f['t2'][key][:] += 1
        f['t3'][key][:] += 1

f = h5py.File('test.h5', 'a', driver='mpio', comm=comm)
chunks = chunk_seq(list(f['t1'].keys()), size)

write_h5(f, chunks[rank])

f.close()

Run it using: mpiexec -n 4 python3 test1.py . The problem is that this will only work if you don't set compression='gzip' while creating the datasets. For reference check the question Does HDF5 support compression with parallel HDF5? If not, why? but I'm not sure if this holds true for the latest version. Looking at this it seems like you'll have to read each dataset serially and create a corresponding dataset in a new HDF5 file with compression.