Pandas：如果为 NaN，则在分组 dataframe 中移动列

Question

I have a grouped dataframe like so:

          │          product          │          category           
          │   spot1   spot2    spot3  │   spot1    spot2    spot3 
──────────┼───────────────────────────┼─────────────────────────────
 basket 1 │   NaN     apple    banana │    NaN     fruits   fruits
 basket 2 │   almond  carrot   NaN    │    nuts    veggies  NaN

One row represents a "basket" containing different food products (vegtables, fruits, nuts).

Each basket has 3 spots that can either contain a food product or not (=NaN).

I would like the first column of group product to be as populated as possible. That means if there is a NaN value in the first column of the product group and some value in the 2nd or n-th column if should shift to the left for each group.

Categories are related: in the example above a baskets' spot1 of group product and spot1 of group category belong together. Every data combination must have a value for product. If product is NaN then all the related items will be NaN as well.

The output should look something like:

          │          product          │          category           
          │   spot1   spot2    spot3  │   spot1    spot2    spot3 
──────────┼───────────────────────────┼─────────────────────────────
 basket 1 │   apple   banana   NaN    │    fruits   fruits   NaN  <-- this row shifted to left to "fill" first spot of product group
 basket 2 │   almond  carrot   NaN    │    nuts    veggies  NaN

---

jezrael's answer here was a good starting point for me:

#for each row remove NaNs and create new Series - rows in final df 
df1 = df.apply(lambda x: pd.Series(x.dropna().values), axis=1)
#if possible different number of columns like original df is necessary reindex
df1 = df1.reindex(columns=range(len(df.columns)))
#assign original columns names
df1.columns = df.columns
print (df1)

However, this solution ignores grouping. I only want values to shift left based on the specific group product .

---

edit / minimal reproducible example

please use this code to get to the "starting point" of problem. The way I get to this point in my production code is more complex but this should do fine.

# Import pandas library
import pandas as pd
  
# initialize list of lists
data = [[1, 'NaN','NaN'], [1, 'apple','fruits'], [1,'banana', 'fruits'], [2, 'carrot','veggies'], [2, 'almond','nuts']]
  
# Create the pandas DataFrame
df = pd.DataFrame(data, columns=['basket','product', 'category'])
  
# print dataframe.
df

dfg = df.groupby(['basket', df.groupby(['basket']).cumcount() + 1]).first().unstack().reset_index()

print(dfg)

Answer 1

I trust there is an easier way to accomplish this, but the following should work.

import pandas as pd
import numpy as np

data = {('product', 'spot1'): {'basket 1': np.nan, 'basket 2': 'almond'}, 
        ('product', 'spot2'): {'basket 1': 'apple', 'basket 2': 'carrot'}, 
        ('product', 'spot3'): {'basket 1': 'banana', 'basket 2': np.nan}, 
        ('category', 'spot1'): {'basket 1': np.nan, 'basket 2': 'nuts'}, 
        ('category', 'spot2'): {'basket 1': 'fruits', 'basket 2': 'veggies'}, 
        ('category', 'spot3'): {'basket 1': 'fruits', 'basket 2': np.nan}}

df = pd.DataFrame(data)

out = df.unstack().dropna()

out.index = pd.MultiIndex.from_arrays([
    out.index.get_level_values(0),
    ('spot' + out.groupby(level=[2,0]).cumcount().add(1).astype(str).to_numpy()),
    out.index.get_level_values(2)])

out = out.reset_index(drop=False).pivot(index='level_2', 
                                        columns=['level_0','level_1'],
                                        values=0)\
    .reindex(df.columns, axis='columns').rename_axis(None, axis=0)

print(out)

         product               category               
           spot1   spot2 spot3    spot1    spot2 spot3
basket 1   apple  banana   NaN   fruits   fruits   NaN
basket 2  almond  carrot   NaN     nuts  veggies   NaN

Explanation

• First, we use df.unstack() with Series.dropna to get a Series with a MultiIndex that consists of col level 0 , col level 1 , index . Ie:

out = df.unstack().dropna()
print(out.head(4))

product   spot1  basket 2    almond
          spot2  basket 1     apple
                 basket 2    carrot
          spot3  basket 1    banana

• Next, we use df.groupby on levels 0,2 (ie original col level 1 and index ), and we use cumcount to get consecutive numbers for the items in each group. We add one and turn the result into a string with add(1).astype(str) and prefix "spot". Ie we are doing:

print(('spot' + df.unstack().dropna().groupby(level=[2,0])\
       .cumcount().add(1).astype(str).to_numpy()))

['spot1' 'spot1' 'spot2' 'spot2' 'spot1' 'spot1' 'spot2' 'spot2']

• We use this result inside pd.MultiIndex.from_arrays to overwrite the MultiIndex (specifically level 1 ) with a new index. Ie we now have:

print(out.head(4))

product  spot1  basket 2    almond
                basket 1     apple
         spot2  basket 2    carrot
                basket 1    banana

• Now, finally, we can reset the index and use df.pivot to change the shape of out so that it matches the shape of the original df . Chaining df.reindex applied to the columns will both reset the order of the columns and add all the missing columns (eg spot3 for both values in col level 0 , and they will be automatically filled with NaNs.