由于文件关闭不当导致的 h5py OSerror

Question

I have some code that runs a simulation for a certain number of time points, and repeats the process a certain number of times. To save my data I use a h5py dataset that i create in lines 10-11. The I run the calculation a certain number of times inside the for loop, and write to the h5py file. This seems to work fine on my computer for time_points = np.arange(10**1) and n_simulations = 10**1 . In line 23, gillespie_ssa is some other function that produces an array as an output and writes it to the appropriate location in the h5py file.

def prod_out(k):
    args = np.array(k)
    time_points = np.arange(10**1)
    n_simulations = 10**1 # no. of simulsations
    nch = k[2] 
    fval = int(k[0]/k[1])

    with h5py.File("./pops%d.hdf5" %fval, "a") as locals()['hdf5_store_{}'.format(fval)]:
        locals()['pops_{}'.format(fval)] = locals()['hdf5_store_{}'.format(fval)].require_dataset('pops_{}'.format(fval),(n_simulations, len(time_points),2), dtype='int8' ,compression="gzip", chunks=True)
    # Run the calculations 

    #locals()['hdf5_store_{}'.format(fval)]= h5py.File("./pops%d.hdf5" %fval, "a") 
    #locals()['pops_{}'.format(fval)] = locals()['hdf5_store_{}'.format(fval)].require_dataset('pops_{}'.format(fval),(n_simulations, len(time_points),2), dtype='int8' ,compression="gzip", chunks=True)
    # Run the calculations 
    for i in range(n_simulations):
        x=np.random.randint(0,k[2]) #start each simulation from random initial condition
        population_0 = np.array([x, k[2]-x])

        with h5py.File("./pops%d.hdf5" %fval, "a") as locals()['hdf5_store_{}'.format(fval)]:
            locals()['pops_{}'.format(fval)] = locals()['hdf5_store_{}'.format(fval)].require_dataset('pops_{}'.format(fval),(n_simulations, len(time_points),2), dtype='int8' ,compression="gzip", chunks=True)
            locals()['pops_{}'.format(fval)][i,:,:]=(gillespie_ssa(propensity, update,
                                    population_0, time_points, fval ,args=args))

    return locals()['hdf5_store_{}'.format(fval)].close()

I was then running it on my university's cluster time_points = np.arange(10**12) and n_simulations = 10**3 , however, my job was timed out, and since the program was interrupted, the files i produced, when I tried to read them give the following error:

OSError: Unable to open file (truncated file: eof = 96, sblock->base_addr = 0, stored_eof = 2048)

I believe this was due to improper closing of hdf5 file? I thought commenting out lines 14 and 15, and using with statement would solve this problem, however, i still get the same error in my test cases when I interrupt the job. (It yields output if I allow it to finish properly). Does the with statement require explicit .flush() of .close() ; is this what I am missing?

What is a safe way to read and write using h5py file, and to ensure that there is still usable data in the file even if there is an interruption?

Answer 1

You may split your job into the smaller portions, saving every part into a different file, finally, merging the files into one large one in the end.

When running your script you have to check which parts are already finished, and skip doing them again, then eventually you'll come to the completion.

And answering your question: No, hdf5 files are not safely saved when the program termination happens, you have to close them properly yourself.