将日期字符串转换为pandas时间序列索引的最有效方法

Question

My CSV data file contains dates in the following format:

In: data["DayIndex"].unique()

Out: array(['04/23/17', '04/20/17', '04/21/17', '04/24/17', '04/22/17',
       '05/02/17', '04/27/17', '05/06/17', '04/30/17', '04/25/17',
       '04/26/17', '05/04/17'], dtype=object)    

I want to turn it into a proper pandas time series. I've tried this:

data["DayIndex"] = pandas.DatetimeIndex(data["Day"])

It takes ages even for a few hundred thousand rows. What are my options to speed up the parsing?

Answer 1

data['DayIndex'] = pandas.to_datetime(data['Day'])

Incorporating @ayhan's comment

data['DayIndex'] = pandas.to_datetime(data['Day'], format='%m/%d/%Y')

Or when you import from csv, include parse_dates

data = read_csv(..., parse_dates=['Day'],
     date_parser=lambda s: pandas.datetime.strptime(s, '%m/%d/%y'))

---

Not sure if this became incorporated into the most recent version of pandas... I don't think so, at least I didn't see it in the "What's New" section...

Anyway, we can build a custom parser to reuse old parsings instead of reparsing the same dates we've already seen.

Let's use map and some clever hashing.

# let u be unique date stings.  We'll do this so that we only parse them once.
u = pd.unique(data['Day'])

# then build a dictionary of these
m = dict(zip(u, pd.to_datetime(u, format='%m/%d/%Y')))

# then use `map` to build the new column
data['DayIndex'] = data['Day'].map(m)

Timing

a = np.random.choice(
    ['04/23/17', '04/20/17', '04/21/17', '04/24/17', '04/22/17',
     '05/02/17', '04/27/17', '05/06/17', '04/30/17', '04/25/17',
     '04/26/17', '05/04/17'],
    100000)

data = pd.DataFrame(dict(Day=a))


%%timeit
u = pd.unique(a)
m = dict(zip(u, pd.to_datetime(u, format='%m/%d/%y'))) 
data['Day'].map(m)
100 loops, best of 3: 15.4 ms per loop

%timeit pd.to_datetime(data['Day'], format='%m/%d/%y')
1 loop, best of 3: 206 ms per loop