日期范围函数的矢量化，用于转换熊猫数据框

Question

This is a problem of transforming date range to a numerical values based on the current date.

Input table:

   ID   START_DATE  END_DATE    CURRENT_DATE
    1   2010-12-08  2011-03-01  2011-04-01
    2   2010-12-10  2011-01-12  2011-01-02
    3   2010-12-16  2011-03-07  2010-10-10

Output table:

   ID   START_DATE  END_DATE    CURRENT_DATE    number_of_days
   1    2010-12-08  2011-03-01  2011-04-01      78.148490
   2    2010-12-10  2011-01-12  2011-01-02      23.726149
   3    2010-12-16  2011-03-07  2010-10-10      0.000000

where nubmer_of_days is computed based on an exponential decay function, followed by summation of all values for one row.

We can implement a function as follows:

def transform(start, end, current):
    value = 0
    if current > end: #current date is later than the end date
        delta = end - start 
        for i in range(delta.days + 1):
            diff = current - (start + td(days = i))
            value += math.exp(- 0.001 * diff.days)
    elif current > start: #current date is between the start and end
        delta = current - start
        for i in range(delta.days + 1):
            diff = current - (start + td(days = i))
            value += math.exp(-0.001 * diff.days)
    else:
        pass
    return value

and then apply the below transformation:

df['number_of_days'] = df.apply(lambda x: transform(x['START_DATE'], x['END_DATE'], x['CURRENT_DATE']),axis=1)

However, this is very slow for a table with millions of rows and huge date range.

Any idea on how to speed up the process by vectorizing the inner for loop in the transformation function?

Thank you!

Answer 1

You could vectorize using numpy array functions to calculate the exponential decay.

df = df[df.CURRENT_DATE > df.START_DATE] # just focusing on cases with calculation

Get the relevant delta depending on CURRENT_DATE and END_DATE :

delta = df[['END_DATE', 'CURRENT_DATE']].min(axis=1).subtract(df.START_DATE).dt.days.add(1)

Calculate the shift of the arange() for exponential decay as max of difference between END_DATE and CURRENT_DATE or 0 :

shift = df.CURRENT_DATE.subtract(df.END_DATE).dt.days.clip(lower=0)

Produce and process the (adjusted) arange objects using np.exp() and np.sum() :

df['number_of_days'] = [np.sum(np.exp(-0.001 * (np.arange(d) + s))) for d, s in zip(delta.values, shift.values)]

to get:

   START_DATE   END_DATE CURRENT_DATE  number_of_days
ID                                                   
1  2010-12-08 2011-03-01   2011-04-01       78.148490
2  2010-12-10 2011-01-12   2011-01-02       23.726149

If you compare performance, you see the efficiency gains from saving on loops:

df_test = pd.concat([df for _ in range(100000)])

def transform1(df):
    df = df[df.CURRENT_DATE > df.START_DATE]
    delta = df[['END_DATE', 'CURRENT_DATE']].min(axis=1).subtract(df.START_DATE).dt.days.add(1)
    shift = df.CURRENT_DATE.subtract(df.END_DATE).dt.days.clip(lower=0)
    return [np.sum(np.exp(-0.001 * (np.arange(d) + s))) for d, s in zip(delta.values, shift.values)]

%timeit transform1(df_test)
1 loop, best of 3: 4.99 s per loop

def transform2(df):
    df['end'] = [d.days for d in df.CURRENT_DATE - df.START_DATE]
    df['start'] = (df.end - [max(0, d.days + 1) for d in (df.END_DATE.where(df.CURRENT_DATE > df.END_DATE, df.CURRENT_DATE) - df.START_DATE)])
    df['number_of_days'] = [sum(np.exp(-0.001 * i) for i in np.arange(stop, start, -1)) for start, stop in zip(df.start, df.end)]
    df.drop(['start', 'end'], axis=1, inplace=True)

%timeit transform2(df_test)
1 loop, best of 3: 36.7 s per loop

Answer 2

You want to get the start and end (integers) for each date range. Then it is relatively easy to vectorize the number_of_days calculation.

df['end'] = [d.days for d in df.CURRENT_DATE - df.START_DATE]
df['start'] = (
    df.end - [max(0, d.days + 1) 
              for d in (df.END_DATE.where(df.CURRENT_DATE > df.END_DATE, df.CURRENT_DATE) 
                        - df.START_DATE)])

df['number_of_days'] = [sum(np.exp(-0.001 * i) for i in np.arange(stop, start, -1)) 
                        for start, stop in zip(df.start, df.end)]
df.drop(['start', 'end'], axis=1, inplace=True)

>>> df
   ID START_DATE   END_DATE CURRENT_DATE  number_of_days
0   1 2010-12-08 2011-03-01   2011-04-01       78.148490
1   2 2010-12-10 2011-01-12   2011-01-02       23.726149
2   3 2010-12-16 2011-03-07   2010-10-10        0.000000