熊猫dropna在寻找均值方面未按预期工作

Question

When I run the code below I get the error:

TypeError: 'NoneType' object has no attribute ' getitem '

    import pyarrow 
    import pandas
    import pyarrow.parquet as pq

    df = pq.read_table("file.parquet").to_pandas()
    df = df.iloc[1:,:]
    df = df.dropna (how="any", inplace = True) # modifies it in place, creates new dataset without NAN

    average_age = df["_c2"].mean()
    print average_age

The dataframe looks like this:

         _c0     _c1  _c2    
    0  RecId   Class  Age   
    1      1      1st   29   
    2      2      1st   NA   
    3      3      1st   30  

If I print the df after calling the dropna method, I get 'None'.

Shouldn't it be creating a new dataframe without the 'NA' in it, which would then allow me to get the average age without throwing an error?

Answer 1

As per OP's comment, the NA is a string rather than NaN. So dropna() is no good here. One of many possible options for filtering out the string value 'NA' is:

df = df[df["_c2"] != "NA"]

A better option to catch inexact matches (eg with trailing spaces) as suggested by @DJK in the comments:

df = df[~df["_c2"].str.contains('NA')]

This one should remove any strings rather than only 'NA':

df = df[df[“_c2”].apply(lambda x: x.isnumeric())]

Answer 2

This will work, also if you the NA in your df is NaN (np.nan), this will not affect your getting the mean of the column, only if your NA is 'NA', which is string

(df.apply(pd.to_numeric,errors ='coerce',axis=1)).describe()
Out[9]: 
       _c0  _c1        _c2
count  3.0  0.0   2.000000
mean   2.0  NaN  29.500000
std    1.0  NaN   0.707107
min    1.0  NaN  29.000000
25%    1.5  NaN  29.250000
50%    2.0  NaN  29.500000
75%    2.5  NaN  29.750000
max    3.0  NaN  30.000000

More info

df.apply(pd.to_numeric,errors ='coerce',axis=1)# all object change to NaN and will not affect getting mean
Out[10]: 
   _c0  _c1   _c2
0  NaN  NaN   NaN
1  1.0  NaN  29.0
2  2.0  NaN   NaN
3  3.0  NaN  30.0