fit_transform中的错误：输入包含NaN，无穷大或对于dtype（'float64'）而言太大的值

Question

I have a dataframe of shape (14407, 2564). I am trying to remove low variance features using the VarianceThreshold function. However, when I call fit_transform, I get the following error:

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

Before usign VarianceThreshold, I replaces all the missing value from my df using the below code:

    df.replace('null',np.NaN, inplace=True)
    df.replace(r'^\s*$', np.NaN, regex=True, inplace=True)
    df.fillna(value=df.median(), inplace=True)

I checked my dataframe afterwards for any empty/infinite values using:

    m = df.isnull().any()
    print "========= COLUMNS WITH NULL VALUES ================="
    print m[m]
    print "========= COLUMNS WITH INFINITE VALUES ================="
    m = np.isfinite(df.select_dtypes(include=['float64'])).any()
    print m[m]

and I got an empty Series as an output, which means all my columns do not have any missing values. The output was:

    ========= COLUMNS WITH NULL VALUES =================
    Series([], dtype: bool)
    ========= COLUMNS WITH INFINITE VALUES =================
    Series([], dtype: bool)

Full error trace:

    Traceback (most recent call last):
      File "/home/users/MyUsername/MyProject/src/main/python/Main.py", line 222, in <module>
        main()
      File "/home/users/MyUsername/MyProject/src/main/python/Main.py", line 218, in         main
        getAllData()
      File "/home/users/MyUsername/MyProject/src/main/python/Main.py", line 95, in getAllData
        predictors, labels, dropped_features = fselector.process(variance=True, corr=True, bestf=True, bestfk=200)
      File         "/home/users/MyUsername/MyProject/src/main/python/classes/featureselector.py", line 54, in process
        self.getVariance(threshold=(.95 * (1 - .95)))
      File "/home/users/MyUsername/MyProject/src/main/python/classes/featureselector.py", line 136, in getVariance
        self.removeLowVarianceColumns(df=self.X, thresh=threshold)
      File "/home/users/MyUsername/MyProject/src/main/python/classes/featureselector.py", line 213, in removeLowVarianceColumns
        selector.fit_transform(df)
      File "/usr/lib64/python2.7/site-packages/sklearn/base.py", line 494, in fit_transform
        return self.fit(X, **fit_params).transform(X)
      File "/usr/lib64/python2.7/site-packages/sklearn/feature_selection/variance_threshold.py", line 64, in fit
        X = check_array(X, ('csr', 'csc'), dtype=np.float64)
    File "/usr/lib64/python2.7/site-packages/sklearn/utils/validation.py", line 407, in check_array
        _assert_all_finite(array)
    File "/usr/lib64/python2.7/site-packages/sklearn/utils/validation.py", line 58, in _assert_all_finite
    " or a value too large for %r." % X.dtype)
    ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

So, I am not sure what to check, this does not seem like a missing value issue, but I am also not able to get what columns/values are causing the problem.

I've seen several threads here that all end in having a missing value, but that does not seem to be the problem here.

Answer 1

I solved this by casting my data to numeric. It appears that, although the error message states 'float64', my data was all objects only and objects did not work well with fit_transform.

Changing my data to float using: df = df.apply(lambda x: pd.to_numeric(x,errors='ignore')) solved the issue.