Fill all values in a group with the first non-null value in that group

Question

The following is the pandas dataframe I have:

cluster Value
1         A
1        NaN
1        NaN
1        NaN
1        NaN
2        NaN
2        NaN
2         B
2        NaN
3        NaN
3        NaN
3         C
3        NaN
4        NaN
4         S
4        NaN
5        NaN
5         A
5        NaN
5        NaN

If we look into the data, cluster 1 has Value 'A' for one row and remain all are NA values. I want to fill 'A' value for all the rows of cluster 1. Similarly for all the clusters. Based on one of the values of the cluster, I want to fill the remaining rows of the cluster. The output should be like,

cluster Value
1         A
1         A
1         A
1         A
1         A
2         B
2         B
2         B
2         B
3         C
3         C
3         C
3         C
4         S
4         S
4         S
5         A
5         A
5         A
5         A

I am new to python and not sure how to proceed with this. Can anybody help with this ?

Answer 1

groupby + bfill , and ffill

df = df.groupby('cluster').bfill().ffill()
df

    cluster Value
0         1     A
1         1     A
2         1     A
3         1     A
4         1     A
5         2     B
6         2     B
7         2     B
8         2     B
9         3     B
10        3     B
11        3     C
12        3     C
13        4     S
14        4     S
15        4     S
16        5     A
17        5     A
18        5     A
19        5     A

---

Or,

groupby + transform with first

df['Value'] = df.groupby('cluster').Value.transform('first')
df

    cluster Value
0         1     A
1         1     A
2         1     A
3         1     A
4         1     A
5         2     B
6         2     B
7         2     B
8         2     B
9         3     B
10        3     B
11        3     C
12        3     C
13        4     S
14        4     S
15        4     S
16        5     A
17        5     A
18        5     A
19        5     A

Answer 2

Edit

The following seems better:

nan_map = df.dropna().set_index('cluster').to_dict()['Value']
df['Value'] = df['cluster'].map(nan_map)

print(df)

Original

I can't think of a better way to do this than iterate over all the rows, but one might exist. First I built your DataFrame:

import pandas as pd
import math

# Build your DataFrame
df = pd.DataFrame.from_items([
    ('cluster', [1,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,5,5,5,5]),
    ('Value', [float('nan') for _ in range(20)]),
])
df['Value'] = df['Value'].astype(object)
df.at[ 0,'Value'] = 'A'
df.at[ 7,'Value'] = 'B'
df.at[11,'Value'] = 'C'
df.at[14,'Value'] = 'S'
df.at[17,'Value'] = 'A'

Now here's an approach that first creates a nan_map dict, then sets the values in Value as specified in the dict.

# Create a dict to map clusters to unique values
nan_map = df.dropna().set_index('cluster').to_dict()['Value']
# nan_map: {1: 'A', 2: 'B', 3: 'C', 4: 'S', 5: 'A'}

# Apply
for i, row in df.iterrows():
    df.at[i,'Value'] = nan_map[row['cluster']]

print(df)

Output:

cluster Value
0         1     A
1         1     A
2         1     A
3         1     A
4         1     A
5         2     B
6         2     B
7         2     B
8         2     B
9         3     C
10        3     C
11        3     C
12        3     C
13        4     S
14        4     S
15        4     S
16        5     A
17        5     A
18        5     A
19        5     A

Note: This sets all values based on the cluster and doesn't check for NaN-ness. You may want to experiment with something like:

# Apply
for i, row in df.iterrows():
    if isinstance(df.at[i,'Value'], float) and math.isnan(df.at[i,'Value']):
        df.at[i,'Value'] = nan_map[row['cluster']]

to see which is more efficient (my guess is the former, without the checks).