根据两个条件为来自另一个数据帧的数据帧赋值

Question

I am trying to assign values from a column in df2['values'] to a column df1['values']. However values should only be assigned if:

1. df2['category'] is equal to the df1['category'] (rows are part of the same category)
2. df1['date'] is in df2['date_range'] (date is in a certain range for a specific category)

So far I have this code, which works, but is far from efficient, since it takes me two days to process the two dfs (df1 has ca. 700k rows).

for i in df1.category.unique():
for j in df2.category.unique():
    if i == j: # matching categories
        for ia, ra in df1.loc[df1['category'] == i].iterrows():
            for ib, rb in df2.loc[df2['category'] == j].iterrows():
                if df1['date'][ia] in df2['date_range'][ib]:
                    df1.loc[ia, 'values'] = rb['values']
                    break

I read that I should try to avoid using for-loops when working with dataframes. List comprehensions are great, however since I do not have a lot of experience yet, I struggle formulating more complicated code.

How can I iterate over this problem more efficient? What essential key aspect should I think about when iterating over dataframes with conditions?

The code above tends to skip some rows or assigns them wrongly, so I need to do a cleanup afterwards. And the biggest problem, that it is really slow.

Thank you.

Some df1 insight:

df1.head()

    date                          category
0  2015-01-07                       f2
1  2015-01-26                       f2
2  2015-01-26                       f2
3  2015-04-08                       f2
4  2015-04-10                       f2

Some df2 insight:

df2.date_range[0]

DatetimeIndex(['2011-11-02', '2011-11-03', '2011-11-04', '2011-11-05',
               '2011-11-06', '2011-11-07', '2011-11-08', '2011-11-09',
               '2011-11-10', '2011-11-11', '2011-11-12', '2011-11-13',
               '2011-11-14', '2011-11-15', '2011-11-16', '2011-11-17',
               '2011-11-18'],
              dtype='datetime64[ns]', freq='D')

df2 other two columns:

df2[['values','category']].head()

            values             category
0            01                  f1
1            02                  f1
2           2.1                  f1
3           2.2                  f1
4            03                  f1

Answer 1

Edit: Corrected erroneous code and added OP input from a comment

Alright so if you want to join the dataframes on similar categories, you can merge them :

import pandas as pd

df3 = df1.merge(df2, on = "category")

Next, since date is a timestamp and the "date_range" is actually generated from two columns, per OP's comment, we rather use :

mask = (df3["startdate"] <= df3["date"]) & (df3["date"] <= df3["enddate"])

subset = df3.loc[mask]

Now we get back to df1 and merge on the common dates while keeping all the values from df1 . This will create NaN for the subset values where they didn't match with df1 in the earlier merge.

As such, we set df1["values"] where the entries in common are not NaN and we leave them be otherwise.

common_dates = df1.merge(subset, on = "date", how= "left") # keeping df1 values

df1["values"] = np.where(common_dates["values_y"].notna(), 
                         common_dates["values_y"], df1["values"])

NB : If more than one df1["date"] matches with the date range, you'll have to drop some values otherwise duplicates mess up the explanation.

Answer 2

You could accomplish the first point:

1. df2['category'] is equal to the df1['category']

with the use of a join.

You could then use a for loop for filtering the data poings from df1[date] inside the merged dataframe that are not contemplated in the df2[date_range]. Unfortunately I need more information about the content of df1[date] and df2[date_range] to write the code here that would exactly do that.