Reading hdf5 datasets with pandas

Question

I'm trying to open a group-less hdf5 file with pandas:

import pandas as pd
foo = pd.read_hdf('foo.hdf5')

but I get an error:

TypeError: cannot create a storer if the object is not existing nor a value are passed

I tried solving this by assigning a key :

foo = pd.read_hdf('foo.hdf5','key')

which works if key was a group, but the file has no groups, but rather several datasets in the highest hdf structure. ie the structure of the working file is: Groups --> Datasets, while the structure of the not working file is: Datasets. Both work fine when opening them with h5py, where I would use:

f = h5py.File('foo.hdf5','r')

and

dset = f['dataset']

to view a dataset. Any ideas how to read this in pandas?

Answer 1

I think you'are confused by different terminology - Pandas's HDF store key is a full path ie Group + DataSet_name ...

demo:

In [67]: store = pd.HDFStore(r'D:\temp\.data\hdf\test.h5')

In [68]: store.append('dataset1', df)

In [69]: store.append('/group1/sub_group1/dataset2', df)

In [70]: store.groups
Out[70]:
<bound method HDFStore.groups of <class 'pandas.io.pytables.HDFStore'>
File path: D:\temp\.data\hdf\test.h5
/dataset1                              frame_table  (typ->appendable,nrows->9,ncols->2,indexers->[index])
/group1/sub_group1/dataset2            frame_table  (typ->appendable,nrows->9,ncols->2,indexers->[index])>

In [71]: store.items
Out[71]:
<bound method HDFStore.items of <class 'pandas.io.pytables.HDFStore'>
File path: D:\temp\.data\hdf\test.h5
/dataset1                              frame_table  (typ->appendable,nrows->9,ncols->2,indexers->[index])
/group1/sub_group1/dataset2            frame_table  (typ->appendable,nrows->9,ncols->2,indexers->[index])>

In [72]: store.close()

In [73]: x = pd.read_hdf(r'D:\temp\.data\hdf\test.h5', 'dataset1')

In [74]: x.shape
Out[74]: (9, 2)

In [75]: x = pd.read_hdf(r'D:\temp\.data\hdf\test.h5', '/group1/sub_group1/dataset2')

In [76]: x.shape
Out[76]: (9, 2)