根据来自另一个 DataFrame 的标准过滤 Pandas 中的 DataFrame

Question

I have two data frames, one with millions of rows of data and another one with rather only a few hundreds of records and I need to filter the first dataframe by three columns from second.

So basically I need to iterate through each row in df2 and see if there any rows in df1 with same tickers and dates between the start and end date. Sadly, I have no clue how to perform this with python.

So my data frames are similar to the following

     Ticker    date
1    AA       2013-12-31 
3    AA       2015-02-28 
4    AA       2016-03-31 
5    AA       2016-04-30 
6    BB       2014-05-31 
7    BB       2014-06-30 
8    BB       2017-07-31 
9    CC       2014-08-31 
10   CC       2017-09-30 
11   CC       2018-10-31 
12   CC       2018-11-30 
13   DD       2018-11-30 
14   DD       2018-12-21

Second one:

     Ticker    StartDate   EndDate
1    AA       2016-01-01   2017-01-01
2    BB       2014-01-01   2015-01-01
3    CC       2018-01-01   2019-01-01
4    AA       2013-01-01   2014-01-01

My expected result is filtered first data frame with all records for all tickers in df2 between start and end dates:

   Ticker     date
1    AA       2013-12-31  
2    AA       2016-03-31 
3    AA       2016-04-30 
4    BB       2014-05-31 
5    BB       2014-06-30  
6    CC       2018-11-30 

UPD

So i've tried the following:

df4 = pd.DataFrame()
###create empty dataframe
for index, row in df2.iterrows():
    df3 =df1.loc[(df1['DATE']>=row['StartDate'])&(df1['DATE']<=row['EndDate'])&(df1['Ticker'] ==row['Ticker'])]
###Go through rows of dataframe2, for every row i look if there any rows in df1 that falls under criteria 
    df4 = df4.append(df3)
### append filtered results of one row to empty dataframe 

It works but it takes ages - I've tried to filter 2% of my data and it took around 25 minutes

Is there any way to speed it up?

Answer 1

Try this:

df3 = df1.merge(df2)
df3 =df3.loc[(df3['date']>=df3['StartDate'])&(df3['date']<=df3['EndDate'])]
df3.drop(['date'], axis = 1)

Answer 2

It's look like you can use group by from create the date ranges for each Ticker

data = pd.read_clipboard()
flt_df = pd.read_clipboard()

data

   Ticker        date
1      AA  2013-12-31
3      AA  2015-02-28
4      AA  2016-03-31
5      AA  2016-04-30
6      BB  2014-05-31
7      BB  2014-06-30
8      BB  2017-07-31
9      CC  2014-08-31
10     CC  2017-09-30
11     CC  2018-10-31
12     CC  2018-11-30
13     DD  2018-11-30
14     DD  2018-12-21

flt_df

  Ticker   StartDate     EndDate
1     AA  2016-01-01  2017-01-01
2     BB  2014-01-01  2015-01-01
3     CC  2018-01-01  2019-01-01
4     AA  2013-01-01  2014-01-01

grouped_df = flt_df.groupby('Ticker').agg({'StartDate':'min','EndDate':'max'})
merged = data.set_index('Ticker').join(grouped_df)
merged = merged[(merged.date>=merged.StartDate)&(merged.date<=merged.EndDate)]
merged.drop(['StartDate','EndDate'],axis=1,inplace=True)

merged

              date
Ticker            
AA      2013-12-31
AA      2015-02-28
AA      2016-03-31
AA      2016-04-30
BB      2014-05-31
BB      2014-06-30
CC      2018-10-31
CC      2018-11-30