Why can't I read multiple HDF5 files from a folder? Meanwhile, I can read a single HDF5 file fine... (Python 3.7)

Question

I'm having trouble with some code to read multiple HDF5 files stored in a directory. I want to be able to read all of them in, then print one of the datasets stored in the HDF5 file. I have a folder full of identical HDF5 files (same 4 datasets, each dataset has the same shape) but they vary in their data (different values stored in each). Why am I experiencing an error while running this?

import h5py
import numpy as np
import os

directory = '/Users/folder'

# for i in os.listdir(directory):
for i in directory:
#     if i.endswith('.h5'):
        with h5py.File(i, 'r') as data:
            extent = np.array(np.degrees(data['extent']))
        print(extent)

Here is the error from the first code snippet:

OSError: Unable to open file (file read failed: time = Thu May 14 12:46:54 2020
, filename = '/', file descriptor = 61, errno = 21, error message = 'Is a directory', buf = 0x7ffee42433b8, total read size = 8, bytes this sub-read = 8, bytes actually read = 18446744073709551615, offset = 0)

But I can run this just fine on a single HDF5 file...

file = 'file.h5'

data = h5py.File(file,'r')
extent = np.array(np.degrees(data['extent']))

print(extent)

And it outputs exactly what it should be:

[   1.   14.  180. -180.]

Answer 1

for i in directory loops over the characters in the string. So ['/', 'U', 's', ...] . The error is telling you that it opened / but it was a directory, not a file. Your commented-out os.listdir(directory) is on the right track, but the yielded file names need to be appended to the base directory to make the full path. You probably want

for i in os.listdir(directory):
    if i.endswith('.h5'):
        with h5py.File(os.path.join(directory, i)) as data:
            ...

Answer 2

I prefer glob() over os.listdir() . Why? Becuase you can use wildcards with the filename, and include the directory in the search (eliminates the need to concatenate the directory to the filename when you open the file).

• glob.glob() returns a list
• glob.iglob() returns an iterator (I prefer in this situation)

Example above redone with glob:

import glob
for h5f in glob.iglob(directory+'/*.h5'):
    with h5py.File(h5f) as data:
         ...