基于条件切片更改Pandas DataFrame中的单元格

Question

I'm playing around with the Titanic dataset, and what I'd like to do is fill in all the NaN/Null values of the Age column with the median value base on that Pclass .

Here is some data:

train

PassengerId Pclass  Age
0   1   3   22
1   2   1   35
2   3   3   26
3   4   1   35
4   5   3   35
5   6   1   NaN
6   7   1   54
7   8   3   2
8   9   3   27
9   10  2   14
10  11  1   Nan

Here is what I would like to end up with:

PassengerId Pclass  Age
0   1   3   22
1   2   1   35
2   3   3   26
3   4   1   35
4   5   3   35
5   6   1   35
6   7   1   54
7   8   3   2
8   9   3   27
9   10  2   14
10  11  1   35

The first thing I came up with is this - In the interest of brevity I have only included one slice for Pclass equal to 1 rather than including 2 and 3:

Pclass_1 = train['Pclass']==1

train[Pclass_1]['Age'].fillna(train[train['Pclass']==1]['Age'].median(), inplace=True)

As far as I understand, this method creates a view rather than editing train itself (I don't quite understand how this is different from a copy, or if they are analogous in terms of memory -- that is an aside I would love to hear about if possible). I particularly like this Q/A on the topic View vs Copy, How Do I Tell? but it doesn't include the insight I'm looking for.

Looking through Pandas docs I learned why you want to use .loc to avoid this pitfall. However I just can't seem to get the syntax right.

Pclass_1 = train.loc[:,['Pclass']==1]

Pclass_1.Age.fillna(train[train['Pclass']==1]['Age'].median(),inplace=True)

I'm getting lost in indices. This one ends up looking for a column named False which obviously doesn't exist. I don't know how to do this without chained indexing. train.loc[:,train['Pclass']==1] returns an exception IndexingError: Unalignable boolean Series key provided .

Answer 1

In this part of the line,

train.loc[:,['Pclass']==1]

the part ['Pclass'] == 1 is comparing the list ['Pclass'] to the value 1 , which returns False . The .loc[] is then evaluated as .loc[:,False] which is causing the error.

I think you mean:

train.loc[train['Pclass']==1]

which selects all of the rows where Pclass is 1. This fixes the error, but it will still give you the "SettingWithCopyWarning".

EDIT 1

(old code removed)

Here is an approach that uses groupby with transform to create a Series containing the median Age for each Pclass . The Series is then used as the argument to fillna() to replace the missing values with the median. Using this approach will correct all passenger classes at the same time, which is what the OP originally requested. The solution comes from the answer to Python-pandas Replace NA with the median or mean of a group in dataframe

import pandas as pd
from io import StringIO

tbl = """PassengerId Pclass  Age
0   1   3   22
1   2   1   35
2   3   3   26
3   4   1   35
4   5   3   35
5   6   1
6   7   1   54
7   8   3   2
8   9   3   27
9   10  2   14
10  11  1
"""

train = pd.read_table(StringIO(tbl), sep='\s+')
print('Original:\n', train)
median_age = train.groupby('Pclass')['Age'].transform('median') #median Ages for all groups
train['Age'].fillna(median_age, inplace=True)
print('\nNaNs replaced with median:\n', train)

The code produces:

 Original:
     PassengerId  Pclass   Age
0             1       3  22.0
1             2       1  35.0
2             3       3  26.0
3             4       1  35.0
4             5       3  35.0
5             6       1   NaN
6             7       1  54.0
7             8       3   2.0
8             9       3  27.0
9            10       2  14.0
10           11       1   NaN

NaNs replaced with median:
     PassengerId  Pclass   Age
0             1       3  22.0
1             2       1  35.0
2             3       3  26.0
3             4       1  35.0
4             5       3  35.0
5             6       1  35.0
6             7       1  54.0
7             8       3   2.0
8             9       3  27.0
9            10       2  14.0
10           11       1  35.0

One thing to note is that this line, which uses inplace=True :

train['Age'].fillna(median_age, inplace=True)

can be replaced with assignment using .loc :

train.loc[:,'Age'] = train['Age'].fillna(median_age)