从h5文件读取n行

Question

I want to read only 10 rows from an h5 file:

df = pd.read_hdf('data.h5', 'cleanuserbase', start=0, stop=10)

But that doesn't work because it reads all the rows.

Answer 1

This works only if your object is table format (rather than fixed format ).

In [11]: df = pd.DataFrame(np.random.randn(100, 2))

In [12]: store = pd.HDFStore('store.h5')

In [13]: df.to_hdf("store.h5", "df", format="table")

In [14]: store.select("df", "index < 2")
Out[14]:
          0         1
0 -0.245982 -1.047534
1 -0.633943 -1.218812

In [15]: pd.read_hdf("store.h5", "df", start=0, stop=2)  # works if non-integer index
Out[15]:
          0         1
0 -0.245982 -1.047534
1 -0.633943 -1.218812

See table format in the docs.

---

If your table is fixed format it can only be read in whole (but perhaps this should raise ):

In [21]: df.to_hdf("store.h5", "fixed_df", format="fixed")

In [22]: pd.read_hdf("store.h5", "fixed_df", start=0, stop=2)
Out[22]:
           0         1
0   2.532604 -0.084852
1   0.735833 -1.100600
2  -0.415245 -2.050627
3  -0.915045 -0.638667
...  # and all the other rows

Answer 2

This is not implemented for fixed stores ATM (but works for table stores, see Andy's answer), see the open issue here

That said, the stores themselves do actually support indexing. Its just not built out. This is peeking into the internals.

In [35]: df = DataFrame(np.random.randn(10,2),columns=list('ab'))

In [36]: store = pd.HDFStore('test.h5',mode='w')

In [37]: store.put('df',df)

In [38]: store
Out[38]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df            frame        (shape->[10,2])

In [39]: mask = slice(4,10)

In [40]: s = store.get_storer('df').storable

In [41]: DataFrame(s.block0_values[mask],index=s.axis1[mask],columns=s.axis0)
Out[41]: 
axis0         a         b
4     -1.347325 -0.936605
5     -0.342814 -0.452055
6      0.951228  0.160918
7     -0.096133  0.816032
8     -0.731431  1.190047
9     -1.050826  0.348107

In [42]: store.close()

I suppose this could raise NotImplementedError until this issue is resolved.