在 Pandas 中逐行比较一个日期框中的日期列值与另一个数据框中的两个日期列

Question

I have a dataframe like this with two date columns and a quamtity column :

     start_date       end_date          qty
1    2018-01-01      2018-01-08         23
2    2018-01-08      2018-01-15         21           
3    2018-01-15      2018-01-22         5
4    2018-01-22      2018-01-29         12

I have a second dataframe with just column containing yearly holidays for a couple of years, like this:

         holiday
1       2018-01-01 
2       2018-01-27
3       2018-12-25
4       2018-12-26

I would like to go through the first dataframe row by row and assign boolean value to a new column holidays if a date in the second data frame falls between the date values of the first date frame. The result would look like this:

  start_date       end_date          qty         holidays
1    2018-01-01      2018-01-08         23       True
2    2018-01-08      2018-01-15         21       False  
3    2018-01-15      2018-01-22         5        False
4    2018-01-22      2018-01-29         12       True

When I try to do that with a for loop I get the following error:

ValueError: Can only compare identically-labeled Series objects

An answer would be appreciated.

Answer 1

If you want a fully-vectorized solution, consider using the underlying numpy arrays:

import numpy as np


def holiday_arr(start, end, holidays):
    start = start.reshape((-1, 1))
    end = end.reshape((-1, 1))
    holidays = holidays.reshape((1, -1))
    result = np.any(
        (start <= holiday) & (holiday <= end),
        axis=1
    )
    return result

If you have your dataframes as above (calling them df1 and df2 ), you can obtain your desired result by running:

df1["contains_holiday"] = holiday_arr(
    df1["start_date"].to_numpy(),
    df1["end_date"].to_numpy(),
    df2["holiday"].to_numpy()
)

df1 then looks like:

  start_date   end_date  qty  contains_holiday
1 2018-01-01 2018-01-08   23              True
2 2018-01-08 2018-01-15   21             False
3 2018-01-15 2018-01-22    5             False
4 2018-01-22 2018-01-29   12              True

Answer 2

try:

def _is_holiday(row, df2):
    return ((df2['holiday'] >= row['start_date']) & (df2['holiday'] <= row['end_date'])).any()

df1.apply(lambda x: _is_holiday(x, df2), axis=1)

Answer 3

I'm not sure why you would want to go row-by-row. But boolean comparisons would be way faster.

df['holiday'] = ((df2.holiday >= df.start_date) &  (df2.holiday <= df.end_date))

Time

>>> 1000 loops, best of 3: 1.05 ms per loop

Quoting @hchw solution (row-by-row)

def _is_holiday(row, df2):
    return ((df2['holiday'] >= row['start_date']) & (df2['holiday'] <= row['end_date'])).any()

df.apply(lambda x: _is_holiday(x, df2), axis=1)

>>> The slowest run took 4.89 times longer than the fastest. This could mean that an intermediate result is being cached.
100 loops, best of 3: 4.46 ms per loop

Answer 4

Try IntervalIndex.contains with list comprehensiont and np.sum

iix = pd.IntervalIndex.from_arrays(df1.start_date, df1.end_date, closed='both')
df1['holidays'] = np.sum([iix.contains(x) for x in df2.holiday], axis=0) >= 1

Out[812]:
  start_date   end_date  qty  holidays
1 2018-01-01 2018-01-08   23      True
2 2018-01-08 2018-01-15   21     False
3 2018-01-15 2018-01-22    5     False
4 2018-01-22 2018-01-29   12      True

---

Note : I assume start_date , end_date , holiday columns are in datetime format. If they are not, you need to convert them before run above command as follows

df1.start_date = pd.to_datetime(df1.start_date)
df1.end_date = pd.to_datetime(df1.end_date)
df2.holiday = pd.to_datetime(df2.holiday)