如何在另一个列值中具有条件的子组中筛选熊猫数据帧

Question

I am struggling finding a solution, here is the problem.

I have a dataframe of the form:

date         day_time   day_time_counter  area

2019-06-05   morning    1                 1
2019-06-05   morning    1                 2
2019-06-05   morning    1                 3

2019-06-05   morning    2                 1
2019-06-05   morning    2                 2
2019-06-05   morning    2                 3

2019-06-05   morning    3                 1
2019-06-05   morning    3                 3

2019-06-05   evening    1                 1
2019-06-05   evening    1                 2

2019-06-05   evening    2                 1
2019-06-05   evening    2                 2
2019-06-05   evening    2                 3

There are some subgroups per "date", "date_time" and "day_time_counter" (which I separated them with a blank line to make them more visible). Each sub group can have one, two or three "area".

What I want is to filter the df in order to get only one subgroup per "date" and "day_time" which has the largest "day_time_counter" AND contain the 3 different "area" values (1, 2, 3), ie the selected subgroups should contain 3 rows, one per "area" value.

Meaning, after filtering the df above, I should get as OUTPUT:

date         day_time   day_time_counter  area

2019-06-05   morning    2                 1
2019-06-05   morning    2                 2
2019-06-05   morning    2                 3

2019-06-05   evening    2                 1
2019-06-05   evening    2                 2
2019-06-05   evening    2                 3

So far I only managed to filter by getting the subgroup with the largest "day_time_counter" but I do not know how to include the condition of being a complete subgroup with the 3 "area".

df_new = df.sort_values('day_time_counter', ascending=False).drop_duplicates(['area', 'date', 'day_time'])

Thanks a lot for your help!

Answer 1

The following will produce what you're looking for:

area_grp_cols = ["date", "day_time", "day_time_counter"]
counter_grp_cols = ["date", "day_time"]
result = (
    df.assign(area_count=lambda df: df.groupby(area_grp_cols)['area']
                                      .transform("count"))
      .loc[lambda df: df["area_count"] == 3]
      .drop(columns=["area_count"])
      .loc[lambda df: df["day_time_counter"]
                      == df.groupby(counter_grp_cols)["day_time_counter"]
                           .transform("max")]
)

Output:

          date day_time  day_time_counter  area
3   2019-06-05  morning                 2     1
4   2019-06-05  morning                 2     2
5   2019-06-05  morning                 2     3
10  2019-06-05  evening                 2     1
11  2019-06-05  evening                 2     2
12  2019-06-05  evening                 2     3

Answer 2

i think your wanted output should be different (evening have day time 3) so i think my code is correct:

choseing the max that has all 3 areas:

m = df.groupby(['date', 'day_time', 'day_time_counter']).area
new_df = []
for k , _ in m:
    if len(set( _ )) != 3:
        continue
    new_df.append(df[(df.date == k[0]) & (df.day_time == k[1]) & (df.day_time_counter == k[2])])
new_df = pd.concat(new_df, join='outer')

filtering the max daytimes :

g = new_df.groupby(['date', 'day_time'])
g.filter(lambda x: len(set(x.area)) == 3)
g = g.day_time_counter.max()

and wrapping up:

itr = [df[(df.date == idx[0]) & (df.day_time == idx[1]) & (df.day_time_counter == value)] for idx, value in zip(g.index, g)]
new_df = pd.concat(itr, join='outer')
new_df

tell me if this is what you wanted

Answer 3

IIUC:

df['group'] = df['area'].eq(1).cumsum()

df_out = df.groupby(['date','day_time','group'])[['area','day_time_counter']]\
           .agg({'area':lambda x: x.nunique()==3,'day_time_counter':'sum'})

df_out.loc[df_out['area'], 'day_time_counter']\
      .rank(ascending=False, method='dense').eq(1).loc[lambda x: x]\
      .to_frame()\
      .merge(df, on=['date','day_time','group'], suffixes=('_',''))[df.columns]

Output:

   area        date day_time  day_time_counter  group
0     1  2019-06-05  evening                 2      5
1     2  2019-06-05  evening                 2      5
2     3  2019-06-05  evening                 2      5
3     1  2019-06-05  morning                 2      2
4     2  2019-06-05  morning                 2      2
5     3  2019-06-05  morning                 2      2