如何在DataFrame中合并具有值组合的行

Question

I have a DataFrame (df1) as given below

    Hair  Feathers  Legs  Type  Count
 R1  1       NaN     0     1      1
 R2  1        0      Nan   1      32
 R3  1        0      2     1      4
 R4  1       Nan     4     1      27

I want to merge rows based by different combinations of the values in each column and also want to add the count values for each merged row. The resultant dataframe(df2) will look like this:

    Hair  Feathers  Legs  Type  Count
 R1   1      0        0     1     33
 R2   1      0        2     1     36
 R3   1      0        4     1     59

The merging is performed in such a way that any Nan value will be merged with 0 or 1. In df2, R1 is calculated by merging the Nan value of Feathers (df1,R1) with the 0 value of Feathers (df1,R2). Similarly, the value of 0 in Legs (df1,R1) is merged with Nan value of Legs (df1,R2). Then the count of R1 (1) and R2(32) are added. In the same manner R2 and R3 are merged because Feathers value in R2 (df1) is similar to R3 (df1) and Legs value of Nan is merged with 2 in R3 (df1) and the count of R2 (32) and R3 (4) are added.

I hope the explanation makes sense. Any help will be highly appreciated

Answer 1

A possible way to do it is by replicating each of the rows containing NaN and fill them with values for the column.

First, we need to get the possible not-null unique values per column:

unique_values = df.iloc[:, :-1].apply(
       lambda x: x.dropna().unique().tolist(), axis=0).to_dict()   
> unique_values
{'Hair': [1.0], 'Feathers': [0.0], 'Legs': [0.0, 2.0, 4.0], 'Type': [1.0]}

Then iterate through each row of the dataframe and replace each NaN by the possible values for each column. We can do this using pandas.DataFrame.iterrows :

mask = df.iloc[:, :-1].isnull().any(axis=1)

# Keep the rows that do not contain `Nan`
# and then added modified rows

list_of_df = [r for i, r in df[~mask].iterrows()]

for row_index, row in df[mask].iterrows(): 

    for c in row[row.isnull()].index: 

        # For each column of the row, replace 
        # Nan by possible values for the column

        for v in unique_values[c]: 

            list_of_df.append(row.copy().fillna({c:v})) 

df_res = pd.concat(list_of_df, axis=1, ignore_index=True).T

The result is a dataframe where all the NaN have been filled with possible values for the column:

> df_res

   Hair  Feathers  Legs  Type  Count
0   1.0       0.0   2.0   1.0    4.0
1   1.0       0.0   0.0   1.0    1.0
2   1.0       0.0   0.0   1.0   32.0
3   1.0       0.0   2.0   1.0   32.0
4   1.0       0.0   4.0   1.0   32.0
5   1.0       0.0   4.0   1.0   27.0

To get the final result of Count grouping by possible combinations of ['Hair', 'Feathers', 'Legs', 'Type'] we just need to do:

> df_res.groupby(['Hair', 'Feathers', 'Legs', 'Type']).sum().reset_index()  

   Hair  Feathers  Legs  Type  Count
0   1.0       0.0   0.0   1.0   33.0
1   1.0       0.0   2.0   1.0   36.0
2   1.0       0.0   4.0   1.0   59.0

Hope it serves

UPDATE

If one or more of the elements in the row are missing, the procedure looking for all the possible combinations for the missing values at the same time. Let us add a new row with two elements missing:

> df

   Hair  Feathers  Legs  Type  Count
0   1.0       NaN   0.0   1.0    1.0
1   1.0       0.0   NaN   1.0   32.0
2   1.0       0.0   2.0   1.0    4.0
3   1.0       NaN   4.0   1.0   27.0
4   1.0       NaN   NaN   1.0   32.0

We will proceed in similar way, but the replacements combinations will be obtained using itertools.product :

 import itertools 

 unique_values = df.iloc[:, :-1].apply(
       lambda x: x.dropna().unique().tolist(), axis=0).to_dict()

 mask = df.iloc[:, :-1].isnull().any(axis=1) 

 list_of_df = [r for i, r in df[~mask].iterrows()] 

 for row_index, row in df[mask].iterrows():  

     cols = row[row.isnull()].index.tolist() 

     for p in itertools.product(*[unique_values[c] for c in cols]): 

         list_of_df.append(row.copy().fillna({c:v for c, v in zip(cols, p)}))

 df_res = pd.concat(list_of_df, axis=1, ignore_index=True).T       


> df_res.sort_values(['Hair', 'Feathers', 'Legs', 'Type']).reset_index(drop=True)

Hair  Feathers  Legs  Type  Count
1   1.0       0.0   0.0   1.0    1.0
2   1.0       0.0   0.0   1.0   32.0
6   1.0       0.0   0.0   1.0   32.0
0   1.0       0.0   2.0   1.0    4.0
3   1.0       0.0   2.0   1.0   32.0
7   1.0       0.0   2.0   1.0   32.0
4   1.0       0.0   4.0   1.0   32.0
5   1.0       0.0   4.0   1.0   27.0
8   1.0       0.0   4.0   1.0   32.0

> df_res.groupby(['Hair', 'Feathers', 'Legs', 'Type']).sum().reset_index()

   Hair  Feathers  Legs  Type  Count
0   1.0       0.0   0.0   1.0   65.0
1   1.0       0.0   2.0   1.0   68.0
2   1.0       0.0   4.0   1.0   91.0