Append HDF5 文件中的数据

Question

I want to append new date to my already created HDF5 file but I don't how to append more data to it, I don't know the actual syntax for appending

I have created an HDF5 file to save my data in HDF format as

with h5py.File(save_path+'PIC200829_256x256x256x3_fast_sj1.hdf5', 'w') as db:
       db.create_dataset('Predframes', data=trainX)
       db.create_dataset('GdSignal', data=trainY)
# this can create an hdf5 file with given name
# and save the data in the given format 

what I want is that I want to append more data (same data) to it to, in next iteration, instead of overwriting and creating new HDF file, one thing I know that I will change "w" to "a" but I don't know what I need to write for append instead of create

Instead of db.create_dataset('Predframes', data=trainX) as db.append('Predframes', data=trainX) is not the right format/syntax? What should I write to append instead of create?

The shape of the trainX is (2500, 100, 100, 40) so when the next trainX with same shape (2500, 100, 100, 40) is appended with the first one, its size should be (5000, 100, 100, 40) while the size of trainY is (2500,80). After appending it should be (5000, 80)

Answer 1

Here is the required code. The initial creation of the dataset has to specify that the outermost dimension should be able to be resized.

from os import path

def create_for_append(h5file, name, data):
    data = np.asanyarray(data)
    return h5file.create_dataset(
          name, data=data, maxshape=(None,) + data.shape[1:])


filepath = path.join(save_path, 'PIC200829_256x256x256x3_fast_sj1.hdf5')
with h5py.File(filepath, 'w') as db:
    create_for_append(db,'Predframes', trainX)
    create_for_append(db,'GdSignal', trainY)

Then we can append the new data by resizing the dataset and putting the new data in the newly allocated range.

def append_to_dataset(dataset, data):
    data = np.asanyarray(data)
    dataset.resize(len(dataset) + len(data), axis=0)
    dataset[-len(data):] = data


with h5py.File(filepath, 'a') as db:
    append_to_dataset(db['Predframes'], trainX)
    append_to_dataset(db['GdSignal'], trainY)