ufunc isfinite not supported for the input types and the inputs could not be safely coerced to any supported types according to the casting rule safe

Question

I have the following data information.

VIF_X.info()
        <class 'pandas.core.frame.DataFrame'>
    Int64Index: 19831255 entries, 9002631 to 2187136
    Data columns (total 22 columns):
     #   Column                  Dtype         
    ---  ------                  -----         
     0   end_time                datetime64[ns]
     1   end_station             int64         
     2   day_of_week             float64       
     3   business_day            float64       
     4   duration                float64       
     5   Distance_KM             float64       
     6   timezone                int64         
     7   temp                    float64       
     8   dew_point               float64       
     9   pressure                int64         
     10  humidity                int64         
     11  wind_speed              float64       
     12  wind_deg                int64         
     13  wind_gust               float64       
     14  rain_1h                 float64       
     15  rain_3h                 float64       
     16  clouds_all              int64         
     17  End_Station_Region_cat  float64       
     18  weather_main_cat        float64       
     19  end_month_cat           int8          
     20  end_hour                int64         
     21  end_minute              int64         
    dtypes: datetime64[ns](1), float64(12), int64(8), int8(1)
    memory usage: 3.3 GB

When I try the following code

VIF_X = VIF_X.drop(['count'],axis=1)
VIF_X = VIF_X[list(VIF_X.columns)]

vif_info = pd.DataFrame()
vif_info['VIF'] = [variance_inflation_factor(VIF_X.values, i) for i in range(VIF_X.shape[1])]
vif_info['Column'] = VIF_X.columns
vif_info.sort_values('VIF', ascending=False)

I have this error. How can I fix it?

I have already done the test to check if there is any column with infinite data, but I have the result that there is no infinite data, but I still have the following error.

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Answer 1

It looks like the variance_inflation_factor function returns None or np.nan for some inputs.

Depending on your use case you might try either:

VIF_X = VIF_X.drop(['count'],axis=1)
VIF_X = VIF_X[list(VIF_X.columns)]

vif_info = pd.DataFrame()
vif_info['VIF'] = [variance_inflation_factor(VIF_X.values, i) for i in range(VIF_X.shape[1])]
vif_info['Column'] = VIF_X.columns
vif_info.dropna(inplace=True)
vif_info.sort_values('VIF', ascending=False)

or use fillna with the desired value that would mask NaN values (eg -1)