Read a Latex table into a Pandas DataFrame

Question

Is there any easy way to read a Latex table, as generated by the DataFrame method to_latex(), back into another DataFrame?. In particular, I'm looking for something that handles Multiindex. For instance if we have the following file 'test.out':

\begin{tabular}{llllrrr}
\toprule
   &      &     &       1 &       2 &          3 \\
\midrule
a  &  1   & 1.0 &    1898 &    1681 &   1.129090 \\
   &      & 0.1 &    1898 &    1349 &   1.406968 \\
   &  10  & 1.0 &    8965 &    5193 &   1.726362 \\
   &      & 0.1 &    8965 &    1669 &   5.371480 \\
   &  100 & 1.0 &   47162 &   22049 &   2.138963 \\
   &      & 0.1 &   47162 &    5732 &   8.227844 \\
b  &  1   & 1.0 &    8316 &    7200 &   1.155000 \\
   &      & 0.1 &    8316 &    5458 &   1.523635 \\
   &  10  & 1.0 &   43727 &   24654 &   1.773627 \\
   &      & 0.1 &   43727 &    6945 &   6.296184 \\
   &  100 & 1.0 &  284637 &  137391 &   2.071730 \\
   &      & 0.1 &  284637 &   26364 &  10.796427 \\
\bottomrule
\end{tabular}

my first attempt was to read it as

df = pd.read_csv('test.out',
                 sep='&',
                 header=None,
                 index_col=(0,1,2),
                 skiprows=4,
                 skipfooter=3,
                 engine='python')

which does not work correctly since read_csv() picks up the empty fields as new levels of the Multiindex:

In [4]: df.index
Out[4]:
MultiIndex(levels=[[u'       ', u'a      ', u'b      '], [u'      ', u'  1   
', u'  10  ', u'  100 '], [0.1, 1.0]],
       labels=[[1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0], [1, 0, 2, 0, 3, 0, 1, 
0, 2, 0, 3, 0], [1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0]],
       names=[0, 1, 2])

Is there any way to do this?

Answer 1

The astropy module has a LaTeX table reader. But it doesn't support all LaTeX expressions. I had to remove \\toprule, \\midrule, and \\bottomrule. That works for me.

from astropy.table import Table
tab = Table.read('table.tex').to_pandas()

Answer 2

A slightly more complicated solution without astropy would be as follows:

Read in the dataframe without setting an index yet:

df = pd.read_csv('table.tex',
                 sep='&',
                 header=None,
                 skiprows=4,
                 skipfooter=3,
                 engine='python')

Now strip the variable whitespace from the "empty" rows for the first two columns and set these to np.nan :

df.loc[df.loc[:,0].str.strip() == "", 0] = np.nan
df.loc[df.loc[:,1].str.strip() == "", 1] = np.nan

With this, you can use pandas' fillna method and set the columns 0 to 2 as your multi-index:

df = df.fillna(method='ffill', axis=0).set_index([0,1,2])