如何将每列只有 1 个非空条目的 Pandas 数据框中的多行合并为一行？

Question

I am using json_normalize to parse json entries of a pandas column. But, as an output I am getting a dataframe with multiple rows with each row having only one non-null entry. I want to combine all these rows to one row in pandas.

currency    custom.gt   custom.eq   price.gt    price.lt
0   NaN 4.0 NaN NaN NaN
1   NaN NaN NaN 999.0   NaN
2   NaN NaN NaN NaN 199000.0
3   NaN NaN other   NaN NaN
4   USD NaN NaN NaN NaN

Answer 1

You can use ffill (forward fill) and bfill (backfill), which are methods for filling NA values in pandas.

# fill NA values
# option 1: 
df = df.ffill().bfill()

# option 2: 
df = df.fillna(method='ffill').fillna(method='bfill')

print(df)

    currency    custom.gt   custom.eq   price.gt    price.lt
0   USD 4.0 other   999.0   199000.0
1   USD 4.0 other   999.0   199000.0
2   USD 4.0 other   999.0   199000.0
3   USD 4.0 other   999.0   199000.0
4   USD 4.0 other   999.0   199000.0

You can then drop the duplicated rows using drop_duplicates and keep the first one :

df = df.drop_duplicates(keep='first')
print(df)

    currency    custom.gt   custom.eq   price.gt    price.lt
0   USD 4.0 other   999.0   199000.0

Depending on how many times you have to repeat the task, I might also take a look at how the JSON file is structured to see if using a dictionary comprehension could help clean things up so that json_normalize can parse it more easily the first time.

Answer 2

you could do

import pandas as pd
from functools import reduce

df = pd.DataFrame.from_dict({"a":["1", None, None],"b" : [None, None, 1], "c":[None, 3, None]})

def red_func(x,y) :
   if pd.isna(x) or pd.isnull(x) :
     return y
 result = [*map( lambda x : reduce(f,x), [list(row) for i, row in df.iterrows()]),]

Outputs :

In [135]: df
Out[135]:
      a    b    c
0     1  NaN  NaN
1  None  NaN  3.0
2  None  1.0  NaN

In [136]: [*map( lambda x : reduce(f,x), [list(row) for i, row in df.iterrows()]),]
Out[136]: ['1', 3.0, 1.0]