pandas.read_csv：如何在分层索引的 CSV 中将两列解析为日期时间？

Question

I have a CSV file that, simplified, looks like this:

X,,Y,,Z,
Date,Time,A,B,A,B
2017-01-21,01:57:49.390,0,1,2,3
2017-01-21,01:57:50.400,4,5,7,9
2017-01-21,01:57:51.410,3,2,4,1

The first two columns are date and time. When I do"

pandas.read_csv('foo.csv', header=[0,1])

I get the following DataFrame:

            X Unnamed: 1_level_0  Y Unnamed: 3_level_0  Z Unnamed: 5_level_0
         Date               Time  A                  B  A                  B
0  2017-01-21       01:57:49.390  0                  1  2                  3
1  2017-01-21       01:57:50.400  4                  5  7                  9
2  2017-01-21       01:57:51.410  3                  2  4                  1

Ignoring the annoying unnamed entries in the columns for now, I'd like to combine the first two columns into a single datetime. So I tried using the parse_dates argument:

pandas.read_csv('foo.csv', header=[0,1], parse_dates={'datetime': [0,1]})

But all I get from this is a traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 646, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 401, in _read
    data = parser.read()
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 939, in read
    ret = self._engine.read(nrows)
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1585, in read
    names, data = self._do_date_conversions(names, data)
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1364, in _do_date_conversions
    self.index_names, names, keep_date_col=self.keep_date_col)
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 2737, in _process_date_conversion
    data_dict.pop(c)
KeyError: "('X', 'Date')"

I'm not sure why it's hitting a KeyError on ('X', 'Date') , since those are definitely present in the columns. I don't really know if this is a bug in pandas that I should report (I'm using 0.19.2), or if I'm just not understanding something. Any ideas?

Answer 1

You can work around if needed:

import datetime as dt
import pandas as pd

# read in the csv file
df = pd.read_csv('foo.csv', header=[0, 1])

# get a label for the funky column names
date_label, time_label = tuple(df.columns.values)[0:2]

# merge the columns into a single datetime
dates = [
    dt.datetime.strptime('T'.join(ts) + '000', '%Y-%m-%dT%H:%M:%S.%f')
    for ts in zip(df[date_label], df[time_label])]

# save the new column
df['DateTime'] = pd.Series(dates).values

Update:

I have submitted a bug and a pull request for this issue. In response to the bug, jreback (pandas lead maintainer) gave a fairly detailed response about issues with the multi-level header from the example. I think you are already aware of these issues, but you may want to read what he wrote. At the end of the response he had this bit that may provide a work around:

Making a single level is just not useful in a multi-level frame. I would probably do this:

In [25]: pandas.read_csv(StringIO(data), header=0, skiprows=1, parse_dates={'datetime':[0,1]})
Out[25]: 
                 datetime  A  B  A.1  B.1
0 2017-01-21 01:57:49.390  0  1    2    3
1 2017-01-21 01:57:50.400  4  5    7    9
2 2017-01-21 01:57:51.410  3  2    4    1