如何在hdf5中高效保存python pandas数据帧并将其作为R中的数据帧打开？

Question

I think the title covers the issue, but to elucidate:

The pandas python package has a DataFrame data type for holding table data in python. It also has a convenient interface to the hdf5 file format, so pandas DataFrames (and other data) can be saved using a simple dict-like interface (assuming you have pytables installed)

import pandas 
import numpy
d = pandas.HDFStore('data.h5')
d['testdata'] = pandas.DataFrame({'N': numpy.random.randn(5)})
d.close()

So far so good. However, if I then try to load that same hdf5 into RI see things aren't so simple:

> library(hdf5)
> hdf5load('data.h5')
NULL
> testdata
$block0_values
         [,1]      [,2]      [,3]       [,4]      [,5]
[1,] 1.498147 0.8843877 -1.081656 0.08717049 -1.302641
attr(,"CLASS")
[1] "ARRAY"
attr(,"VERSION")
[1] "2.3"
attr(,"TITLE")
[1] ""
attr(,"FLAVOR")
[1] "numpy"

$block0_items
[1] "N"
attr(,"CLASS")
[1] "ARRAY"
attr(,"VERSION")
[1] "2.3"
attr(,"TITLE")
[1] ""
attr(,"FLAVOR")
[1] "numpy"
attr(,"kind")
[1] "string"
attr(,"name")
[1] "N."

$axis1
[1] 0 1 2 3 4
attr(,"CLASS")
[1] "ARRAY"
attr(,"VERSION")
[1] "2.3"
attr(,"TITLE")
[1] ""
attr(,"FLAVOR")
[1] "numpy"
attr(,"kind")
[1] "integer"
attr(,"name")
[1] "N."

$axis0
[1] "N"
attr(,"CLASS")
[1] "ARRAY"
attr(,"VERSION")
[1] "2.3"
attr(,"TITLE")
[1] ""
attr(,"FLAVOR")
[1] "numpy"
attr(,"kind")
[1] "string"
attr(,"name")
[1] "N."

attr(,"TITLE")
[1] ""
attr(,"CLASS")
[1] "GROUP"
attr(,"VERSION")
[1] "1.0"
attr(,"ndim")
[1] 2
attr(,"axis0_variety")
[1] "regular"
attr(,"axis1_variety")
[1] "regular"
attr(,"nblocks")
[1] 1
attr(,"block0_items_variety")
[1] "regular"
attr(,"pandas_type")
[1] "frame"

Which brings me to my question: ideally I would be able to save back and forth from R to pandas. I can obviously write a wrapper from pandas to R (I think... though I think if I use a pandas MultiIndex that might become trickier), but I don't think I can easily then use that data back in pandas. Any suggestions?

Bonus: what I really want to do is use the data.table package in R with a pandas dataframe (the keying approach is suspiciously similar in both packages). Any help on that one greatly appreciated.

Answer 1

If you are still looking at this, take a look at this post on google groups. It shows how to exchange data between pandas/R via HDF5.

https://groups.google.com/forum/?fromgroups#!topic/pydata/0LR72GN9p6w

Answer 2

It would make sense to dropdown to pytables and store/get your data there.

Ultimately a DataFrame is a dict of Series which is what an HDF5 Table is. There are limitations on the translation due to incompatible dtypes but for numerical data it should be straight forward.

The way pandas stores its HDF5 is viewed more like a binary blob. It has to support all the nuances of a DataFrame which HDF5 does support cleanly.

https://github.com/dalejung/trtools/blob/master/trtools/io/pytables.py

Has some that kind of pandas/hdf5 munging code.

Answer 3

如何在HDF5中编写数据帧，以便可以在R中读取它现在在Pandas文档中： http ： //pandas-docs.github.io/pandas-docs-travis/io.html#external-compatibility

Answer 4

I recommend using feather , built by Wes and Hadley to solve the problem of transferring data between R and Python efficiently.

Python

import numpy as np
import pandas as pd
import feather as ft

df = pd.DataFrame({'N': np.random.randn(5)})
ft.write_dataframe(df, 'df.feather')

R

library(data.table)
library(feather)

dt <- data.table(read_feather("df.feather"))
dt
           N
1: 0.2777700
2: 1.4083377
3: 1.2940691
4: 0.8221348
5: 1.8552908

Answer 5

You could use csv files as the common data format. Both R and python pandas can easily work with that. You might lose some precision, but if this is a problem depends on your specific problem.