我该如何更换<NA>在 Pandas DataFrame 中使用 NaN？

Question

Some columns in my DataFrame have instances of <NA> which are of type pandas._libs.missing.NAType .

I'd like to replace them with NaN using np.nan .

I have seen questions where the instances of <NA> can be replaced when using pd.read_csv() .

But since my Pandas DataFrame is created from a Spark DataFrame I do not use the pd.read_csv() function.

Please Advise.

Answer 1

Use replace , but also is necessary upgrade pandas.

df = pd.DataFrame({'age':[pd.NA, 4, 8]})

df = df.replace(pd.NA, np.nan)
print (df)
   age
0  NaN
1  4.0
2  8.0

Answer 2

我对replace解决方案没有任何运气，但能够通过将列转换为 float - df['my_col'].astype(float)将<NA>转换为 np.nan 。

Answer 3

Using Pandas v1.3.1 and Numpy v1.20.3 you can use df.where() which do the replace when the condition is False like below:

$> df = pd.DataFrame({'age':[pd.NA, 4, 8]})
$> print(df)
    age
0  <NA>
1     4
2     8
$> print(type(df.iloc[0]['age']))
   pandas._libs.missing.NAType
$> df = df.where(pd.notnull(df), np.nan)  # Replace pd.NA, np.nan and None by np.nan
$> print(df)
   age
0  NaN
1    4
2    8
$> print(type(df.iloc[0]['age']))
   float

PS: You do also:

$> df = df.where(~pd.isna(df), np.nan)