熊猫填充模式

Question

I have a data set in which there is a column known as Native Country which contain around 30000 records. Some are missing represented by NaN so I thought to fill it with mode() value. I wrote something like this:

data['Native Country'].fillna(data['Native Country'].mode(), inplace=True)

However when I do a count of missing values:

for col_name in data.columns: 
    print ("column:",col_name,".Missing:",sum(data[col_name].isnull()))

It is still coming up with the same number of NaN values for the column Native Country.

Answer 1

Just call first element of series:

data['Native Country'].fillna(data['Native Country'].mode()[0], inplace=True)

or you can do the same with assisgnment:

data['Native Country'] = data['Native Country'].fillna(data['Native Country'].mode()[0])

Answer 2

请注意，NaN 可能是您的数据帧的模式：在这种情况下，您将用另一个 NaN 替换 NaN。

Answer 3

If we fill in the missing values with fillna(df['colX'].mode()) , since the result of mode() is a Series, it will only fill in the first couple of rows for the matching indices. At least if done as below:

fill_mode = lambda col: col.fillna(col.mode())
df.apply(fill_mode, axis=0)

However, by simply taking the first value of the Series fillna(df['colX'].mode()[0]) , I think we risk introducing unintended bias in the data. If the sample is multimodal, taking just the first mode value makes the already biased imputation method worse. For example, taking only 0 if we have [0, 21, 99] as the equally most frequent values. Or filling missing values with False when True and False values are equally frequent in a given column.

I don't have a clear cut solution here. Assigning a random value from all the local maxima could be one approach if using the mode is a necessity.

Answer 4

import numpy as np

import pandas as pd

print(pd.__version__)

1.2.0

df = pd.DataFrame({'Country': [np.nan, 'France', np.nan, 'Spain', 'France'], 'Purchased': [np.nan,'Yes', 'Yes', 'No', np.nan]})

	Country	Purchased
0	NaN	NaN
1	France	Yes
2	NaN	Yes
3	Spain	No
4	France	NaN

 df.fillna(df.mode())  ## only applied on first row because df.mode() returns a dataframe with one row

	Country	Purchased
0	France	Yes
1	France	Yes
2	NaN	Yes
3	Spain	No
4	France	NaN

df = pd.DataFrame({'Country': [np.nan, 'France', np.nan, 'Spain', 'France'], 'Purchased': [np.nan,'Yes', 'Yes', 'No', np.nan]})

df.fillna(df.mode().iloc[0]) ## convert df to a series

	Country	Purchased
0	France	Yes
1	France	Yes
2	France	Yes
3	Spain	No
4	France	Yes

Answer 5

尝试类似： fill_mode = lambda col: col.fillna(col.mode())和函数： new_df = df.apply(fill_mode, axis=0)

Answer 6

You can get the number 'mode' or any another strategy

num = data['Native Country'].mode()
data['Native Country'].fillna(num, inplace=True)

or in one line like this

data['Native Country'].fillna(data['Native Country'].mode()[0], inplace=True)

Answer 7

For those who came here (as I did) to fill NAs in multiple columns, grouped by multiple columns and have problem that mode returns nothing, where there are only NA values in the group:

df[['col_to_fill_NA_1','col_to_fill_NA_2']] = df.groupby(['col_to_group_by_1', 'col_to_group_by_2'], dropna=False)[['col_to_fill_NA_1','col_to_fill_NA_2']].transform(lambda x: x.fillna(x.mode()[0]) if len(x.mode()) == 1 else x)

you can fill any number of "col_to_fill_NA" and make group by any number of "col_to_group_by". The if statement returns mode if mode exists and returns NAs for the groups, where there are only NAs.