Reading multiple hdf5 files from a folder

Question

I currently have a code that imports an hdf5 file, and then computes a function for an area under the curve.

import h5py
file = h5py.File('/Users/hansari/Desktop/blabla', 'r')

xdata = file.get('data')
xdata= np.array(xdata)
xdata_df = pd.DataFrame(xdata)
table = pd.DataFrame(xdata_df).reset_index() 

This is the code I use to fetch the file.

I currently have a folder than has 25 hdf5 files. Is there a way to have it so that I can have the code run all 25 files and spit out the result of the function for all?

I'm hoping to have it import the file, run through the whole script, and then repeat it with the next hdf5 file, instead of importing all the data first and then running through the code with a mass amt of data.

I'm currently using glob.glob , but it's importing all of the files at one go and giving me a huge dataset that is hard to work with.

Answer 1

Without more code, I can't tell you what you are doing wrong. To demonstrate the process, I created a simple example that reads multiple HDF5 files and loads into a Pandas dataframe using glob.iglob() and h5py . See the code below. The table dataframe is created inside the 2nd loop and only contains data from 1 HDF5 file. You should add your function to compute the area under the curve inside the for file in glob.iglob() loop.

# First, create 3 simple H5 files
for fcnt in range(1,4,1):
    fname = f'file_{fcnt}.h5'
    with h5py.File(fname,'w') as h5fw:
        arr = np.random.random(10*10).reshape(10,10)
        h5fw.create_dataset('data',data=arr)

# Loop over H5 files and load into a dataframe
for file in glob.iglob('file*.h5'):   
    with h5py.File(file, 'r') as h5fr:
        xdata = h5fr['data'][()]
        table = pd.DataFrame(xdata).reset_index() 
        print(table)
        # add code to compute area under the curve here