Pandas DataFrame / HDFStore通过CSV传递多种日期格式

Question

I am doing the following to pass multiple dates in different columns. However, the second column ( time ) column doesn't conform to this string so it has errors. How do i achieve this?

 dateparse = lambda x: pd.datetime.strptime(x, '%d/%m/%Y %H:%M:%S')

 for chunk in pd.read_csv(file, chunksize=500000, parse_dates=['date','time'], parse_dates = dateparse, names = col_names, index_col = index_cols, header = 0, dtype = dtype)
        store.append('df',chunk)

Sample Data :

 Date                     Time
19/10/2016 00:00:00      00:05:01

Answer 1

There is no need to specify datetime format if you have a standard format like '19/10/2016 00:00:00' - Pandas will parse it automatically, so you don't need to use date_parser parameter.

Option 1: Converting Time column to datetime64[ns] dtype:

for chunk in pd.read_csv(file, chunksize=500000, parse_dates=['Date'], names=col_names, index_col=index_cols, dtype = dtype):
    chunk['Time'] = chunk['Date'].dt.normalize() + pd.to_timedelta(chunk['Time'])
    store.append('df',chunk)

Option 2: Converting Time column to timedelta64[ns] dtype:

for chunk in pd.read_csv(file, chunksize=500000, parse_dates=['Date'], names=col_names, index_col=index_cols, dtype = dtype):
    chunk['Time'] = pd.to_timedelta(chunk['Time'])
    store.append('df',chunk)

PS both mentioned dtypes are supported by HDFStore

Option 3:

for chunk in pd.read_csv(file, chunksize=500000, names=col_names, index_col=index_cols, dtype = dtype):
    chunk['Date'] = pd.to_datetime(chunk['Date'], errors='coerce')
    chunk['Time'] = pd.to_timedelta(chunk['Time'], errors='coerce')
    store.append('df',chunk)

Answer 2

You can tell Pandas to combine the date and time columns into one column by passing a list of lists instead of just a list in parse_dates as specified in the documentation :

parse_dates : boolean or list of ints or names or list of lists or dict, default False

• boolean. If True -> try parsing the index.
• list of ints or names. eg If [1, 2, 3] -> try parsing columns 1, 2, 3 each as a separate date column.
• list of lists. eg If [[1, 3]] -> combine columns 1 and 3 and parse as a single date column. dict, eg {'foo' : [1, 3]} -> parse columns 1, 3 as date and call result 'foo'

You'll also want to specify dayfirst=True given your date format.

That means your code becomes

for chunk in pd.read_csv(file, chunksize=500000, 
                         parse_dates=[['date', 'time']],  # note the extra []
                         dayfirst=True,
                         names=col_names, index_col=index_cols, 
                         header=0, dtype=dtype)
    store.append('df',chunk)