从Pandas Dataframe中现有的日期和时间列创建datetime列的更快方法

Question

I have a Pandas Dataframe with Year, Month, Day, and Time columns, and I'm trying to combine them into a new column that just has a single datetime object. The data type in each column is an int, including the time column (it goes between 1 and 2359). For example: 2015, 3, 15, 745 would be March 15, 2015 at 7:45AM.

I currently just do this, but it takes several minutes to run on my dataframe that has 58000 rows:

for i in range(len(flights.index)):
    flights['SCHEDULED_DEPARTURE_DATETIME'][i] = datetime.datetime(
        flights.iloc[i]['YEAR'], 
        flights.iloc[i]['MONTH'], 
        flights.iloc[i]['DAY'], 
        int(np.floor(flights.iloc[i]['SCHEDULED_DEPARTURE']/100)), #hours
        flights.iloc[i]['SCHEDULED_DEPARTURE']%100                 #minutes
    ) 

There must be a faster, more pythonic way to do this, but I can't seem to get it to work with apply. What am I missing?

FYI, my dataframe is a small subset of this data set from Kaggle: https://www.kaggle.com/usdot/flight-delays#flights.csv

Answer 1

You could use pd.to_datetime() like this:

import pandas as pd
import numpy as np

data = pd.DataFrame(np.array(
    [
        [2018, 10, 1, 2359],
        [2018, 10, 1, 1500],
        [2018, 10, 1, 900],
        [2018, 10, 1, 1],
        [2018, 10, 1, 0]
    ]
), columns = ['year', 'month', 'day', 'scheduled_departure'])

data['hour'] = np.floor(data['scheduled_departure'] / 100)

data['minute'] = data['scheduled_departure'] % 100

data['scheduled_departure_datetime'] = pd.to_datetime(data[['year', 'month', 'day', 'hour', 'minute']])

print(data['scheduled_departure_datetime'])

Giving:

0   2018-10-01 23:59:00
1   2018-10-01 15:00:00
2   2018-10-01 09:00:00
3   2018-10-01 00:01:00
4   2018-10-01 00:00:00
Name: scheduled_departure_datetime, dtype: datetime64[ns]

I haven't tested speed but I imagine this will be faster.