在 pandas dataframe 中查找特定的值组合

Question

I am preparing a dataframe for machine learning. The data set contains weather data from several weather stations in australia over a period of 10 years. One of the measured attributes is Evaporation. It has about 50% missing values. Now I want to find out, whether the missing values are evenly distributed over all weather stations or if roughly half of the weather stations just never measured Evaporation.

How can I find out about the distribution of a value in combination with another attribute? I basically want to loop over the weather stations and get a count of NaNs and normal values.

rain_df.query('Location == "Albury"').Location.count()

This gives me the number of measurement points from the weaher station in Albury. Now how can I find out how many NaNs were measured in Albury compared to normal (non-NaN) measurements?

Answer 1

You can use .isnull() to mask a series with True for NaNs and False for everything else. Then you can use .value_counts(normalize=True) to get the proportions of NaN and non NaN in that series.

rain_df.query('Location == "Albury"').Location.isnull().value_counts(normalize=True)