Pandas - 在分类数据中填充 NaN

Question

I am trying to fill missing values (NAN) using the below code

NAN_SUBSTITUTION_VALUE = 1
g = g.fillna(NAN_SUBSTITUTION_VALUE)

but I am getting the following error

ValueError: fill value must be in categories.

Would anybody please throw some light on this error.

Answer 1

Your question is missing the important point what g is, especially that it has dtype categorical . I assume it is something like this:

g = pd.Series(["A", "B", "C", np.nan], dtype="category")

The problem you are experiencing is that fillna requires a value that already exists as a category. For instance, g.fillna("A") would work, but g.fillna("D") fails. To fill the series with a new value you can do:

g_without_nan = g.cat.add_categories("D").fillna("D")

Answer 2

Add the category before you fill:

g = g.cat.add_categories([1])
g.fillna(1)

Answer 3

Once you create Categorical Data , you can insert only values in category.

>>> df
    ID  value
0    0     20
1    1     43
2    2     45

>>> df["cat"] = df["value"].astype("category")
>>> df
    ID  value    cat
0    0     20     20
1    1     43     43
2    2     45     45

>>> df.loc[1, "cat"] = np.nan
>>> df
    ID  value    cat
0    0     20     20
1    1     43    NaN
2    2     45     45

>>> df.fillna(1)
ValueError: fill value must be in categories
>>> df.fillna(43)
    ID  value    cat
0    0     20     20
1    1     43     43
2    2     45     45

Answer 4

As many have said before, this error comes from the fact that that feature's type is 'category'.
I suggest converting it to string first, use fillna and finally convert it back to category if needed.

g = g.astype('string')
g = g.fillna(NAN_SUBSTITUTION_VALUE)
g = g.astype('category')

Answer 5

Sometimes you may want to replace the NaN with values present in your dataset, you can use that then:

#creates a random permuation of the categorical values
permutation = np.random.permutation(df[field])

#erase the empty values
empty_is = np.where(permutation == "")
permutation = np.delete(permutation, empty_is)

#replace all empty values of the dataframe[field]
end = len(permutation)
df[field] = df[field].apply(lambda x: permutation[np.random.randint(end)] if pd.isnull(x) else x)

It works quite efficiently.

Answer 6

The deep understanding is because:

Categoricals can only take on only a limited, and usually fixed, number of possible values (categories). In contrast to statistical categorical variables, a Categorical might have an order, but numerical operations (additions, divisions, …) are not possible.

All values of the Categorical are either in categories or np.nan. Assigning values outside of categories will raise a ValueError. Order is defined by the order of the categories, not lexical order of the values.

https://pandas.pydata.org/docs/reference/api/pandas.Categorical.html