为什么h5py数据集并行赋值没有输出？

Question

I am trying to generate an hdf5 dataset from code that runs in parallel, but when I read the dataset generated, it is blank, all entries are zero.

I have replaced the parallel code with a sequential for loop and the dataset works out fine in this case, but I don't know what is the problem with doing the same in parallel.

Here is the code to a minimal example

import h5py
import scipy.stats as st

file = h5py.File('test.hdf5','a')

dset = file.create_dataset('x', (10,1024), maxshape=(None,1024), 
                           dtype='float32')

def assign(j):
    dset[j,:] = st.norm.rvs(0.,1.,1024)

from joblib import Parallel, delayed
import multiprocessing as mp

Parallel(n_jobs=4)(delayed(assign)(j) for j in range(10))

file.close()

And the file is later read with

import h5py

file = h5py.File('test.hdf5','r')
file['x'][:]

What is the issue with the code running in parallel?

Answer 1

Every thread of your parallel code gets it's own copy of the dset and they keep stepping on each other's toes. You may try something like this to get it working:

def get_row(x) :
    return st.norm.rvs(0.,1.,1024)

dset[:,:] = Parallel(n_jobs=4)(delayed(get_row)(j) for j in range(10))

ps. thanks to @gapollo for the correction!