如何在熊猫中将列转换为一个日期时间列？

Question

I have a dataframe where the first 3 columns are 'MONTH', 'DAY', 'YEAR'

In each column there is an integer. Is there a Pythonic way to convert all three columns into datetimes while there are in the dataframe?

From:

M    D    Y    Apples   Oranges
5    6  1990      12        3
5    7  1990      14        4
5    8  1990      15       34
5    9  1990      23       21

into:

Datetimes    Apples   Oranges
1990-6-5        12        3
1990-7-5        14        4
1990-8-5        15       34
1990-9-5        23       21

Answer 1

In version 0.18.1 you can use to_datetime , but:

• The names of the columns have to be year , month , day , hour , minute and second :
• Minimal columns are year , month and day

Sample:

import pandas as pd

df = pd.DataFrame({'year': [2015, 2016],
                   'month': [2, 3],
                    'day': [4, 5],
                    'hour': [2, 3],
                    'minute': [10, 30],
                    'second': [21,25]})

print df
   day  hour  minute  month  second  year
0    4     2      10      2      21  2015
1    5     3      30      3      25  2016

print pd.to_datetime(df[['year', 'month', 'day']])
0   2015-02-04
1   2016-03-05
dtype: datetime64[ns]

print pd.to_datetime(df[['year', 'month', 'day', 'hour']])
0   2015-02-04 02:00:00
1   2016-03-05 03:00:00
dtype: datetime64[ns]

print pd.to_datetime(df[['year', 'month', 'day', 'hour', 'minute']])
0   2015-02-04 02:10:00
1   2016-03-05 03:30:00
dtype: datetime64[ns]

print pd.to_datetime(df)
0   2015-02-04 02:10:21
1   2016-03-05 03:30:25
dtype: datetime64[ns]

Another solution is convert to dictionary :

print df
   M  D     Y  Apples  Oranges
0  5  6  1990      12        3
1  5  7  1990      14        4
2  5  8  1990      15       34
3  5  9  1990      23       21

print pd.to_datetime(dict(year=df.Y, month=df.M, day=df.D))
0   1990-05-06
1   1990-05-07
2   1990-05-08
3   1990-05-09
dtype: datetime64[ns]

Answer 2

In 0.13 (coming very soon), this is heavily optimized and quite fast (but still pretty fast in 0.12); both orders of magnitude faster than looping

In [3]: df
Out[3]: 
   M  D     Y  Apples  Oranges
0  5  6  1990      12        3
1  5  7  1990      14        4
2  5  8  1990      15       34
3  5  9  1990      23       21

In [4]: df.dtypes
Out[4]: 
M          int64
D          int64
Y          int64
Apples     int64
Oranges    int64
dtype: object

# in 0.12, use this
In [5]: pd.to_datetime((df.Y*10000+df.M*100+df.D).apply(str),format='%Y%m%d')

# in 0.13 the above or this will work
In [5]: pd.to_datetime(df.Y*10000+df.M*100+df.D,format='%Y%m%d')
Out[5]: 
0   1990-05-06 00:00:00
1   1990-05-07 00:00:00
2   1990-05-08 00:00:00
3   1990-05-09 00:00:00
dtype: datetime64[ns]

Answer 3

Here is a alternative which uses NumPy datetime64 and timedelta64 arithmetic . It appears to be a bit faster for small DataFrames and much faster for larger DataFrames:

import numpy as np
import pandas as pd

df = pd.DataFrame({'M':[1,2,3,4], 'D':[6,7,8,9], 'Y':[1990,1991,1992,1993]})
#    D  M     Y
# 0  6  1  1990
# 1  7  2  1991
# 2  8  3  1992
# 3  9  4  1993

y = np.array(df['Y']-1970, dtype='<M8[Y]')
m = np.array(df['M']-1, dtype='<m8[M]')
d = np.array(df['D']-1, dtype='<m8[D]')
dates2 = pd.Series(y+m+d)
# 0   1990-01-06
# 1   1991-02-07
# 2   1992-03-08
# 3   1993-04-09
# dtype: datetime64[ns]

---

In [214]: df = pd.concat([df]*1000)

In [215]: %timeit pd.to_datetime((df['Y']*10000+df['M']*100+df['D']).astype('int'), format='%Y%m%d')
100 loops, best of 3: 4.87 ms per loop

In [216]: %timeit pd.Series(np.array(df['Y']-1970, dtype='<M8[Y]')+np.array(df['M']-1, dtype='<m8[M]')+np.array(df['D']-1, dtype='<m8[D]'))
1000 loops, best of 3: 839 µs per loop

---

Here's a helper function to make this easier to use:

def combine64(years, months=1, days=1, weeks=None, hours=None, minutes=None,
              seconds=None, milliseconds=None, microseconds=None, nanoseconds=None):
    years = np.asarray(years) - 1970
    months = np.asarray(months) - 1
    days = np.asarray(days) - 1
    types = ('<M8[Y]', '<m8[M]', '<m8[D]', '<m8[W]', '<m8[h]',
             '<m8[m]', '<m8[s]', '<m8[ms]', '<m8[us]', '<m8[ns]')
    vals = (years, months, days, weeks, hours, minutes, seconds,
            milliseconds, microseconds, nanoseconds)
    return sum(np.asarray(v, dtype=t) for t, v in zip(types, vals)
               if v is not None)

In [437]: combine64(df['Y'], df['M'], df['D'])
Out[437]: array(['1990-01-06', '1991-02-07', '1992-03-08', '1993-04-09'], dtype='datetime64[D]')

Answer 4

I re-approached the problem and I think I found a solution. I initialized the csv file in the following way:

pandas_object = DataFrame(read_csv('/Path/to/csv/file', parse_dates=True, index_col = [2,0,1] ))

Where the:

index_col = [2,0,1]

represents the columns of the [year, month, day]

Only problem now is that now I have three new index columns, one represent the year, another the month, and another the day.

Answer 5

Convert the dataframe to strings for easy string concatenation:

df=df.astype(str)

then convert to datetime, specify the format:

df.index=pd.to_datetime(df.Y+df.M+df.D,format="%Y%m%d")

which replaces the index rather than creating a new column.

Answer 6

Even better way to do is as below:

import pandas as pd

import datetime

dataset = pd.read_csv('dataset.csv')

date=dataset.apply(lambda x: datetime.date(int(x['Yr']), x['Mo'], x['Dy']),axis=1)

date = pd.to_datetime(date)

dataset = dataset.drop(columns=['Yr', 'Mo', 'Dy'])

dataset.insert(0, 'Date', date)

dataset.head()

---

Answer 7

 [pd.to_datetime(str(a)+str(b)+str(c),
                 format='%m%d%Y'
                ) for a,b,c in zip(df.M, df.D, df.Y)]

Answer 8

Let's assume you've got a dictionary foo with each column of dates in parallel. If so, here's your one liner:

>>> from datetime import datetime
>>> foo = {"M": [1,2,3], "D":[30,30,21], "Y":[1980,1981,1982]}
>>>
>>> df = pd.DataFrame({"Datetime": [datetime(y,m,d) for y,m,d in zip(foo["Y"],foo["M"],foo["D"])]})

The real guts of it are this bit:

>>> [datetime(y,m,d) for y,m,d in zip(foo["Y"],foo["M"],foo["D"])]
[datetime.datetime(1980, 1, 30, 0, 0), datetime.datetime(1981, 2, 28, 0, 0), datetime.datetime(1982, 3, 21, 0, 0)]

This is the sort of thing zip was made for. It takes parallel lists and turns them into tuples. Then they get tuple unpacked (the for y,m,d in bit) by the list comprehension there, then fed into the datetime object constructor.

pandas seems happy with the datetime objects.