Getting Nan, Infinite error in sklearn
Question
I am running few ml algos from sklearn. But for all those I am getting the following error
/Users//anaconda/lib/python2.7/site-packages/sklearn/utils/validation.pyc in check_X_y(X, y, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric)
448 else:
449 y = column_or_1d(y, warn=True)
--> 450 _assert_all_finite(y)
451 if y_numeric and y.dtype.kind == 'O':
452 y = y.astype(np.float64)
/Users//anaconda/lib/python2.7/site-packages/sklearn/utils/validation.pyc in _assert_all_finite(X)
50 and not np.isfinite(X).all()):
51 raise ValueError("Input contains NaN, infinity"
---> 52 " or a value too large for %r." % X.dtype)
53
54
ValueError: Input contains NaN, infinity or a value too large for dtype('float64')
Please note that my design matrix has no nan or infinite value. Here is what I did to check:
np.isfinite(X_cohort_pr).all()
Out[259]:
True
X.isnull().any().any()
Out[261]:
False
So if you see my data matrix has no null or infinite values. then why I am getting this error and how to resolve this?. This has taken me more than 8 hours in debugging it.Please help
EDIT2:
Here is the first five rows of the data matrix. It has total 800K rows and some 180 odd features.
array([[ 1. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 1. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 1. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 1. , 0. , 0. , 0. , 0. ,
0. , 1. , 0. , 0. , 0. , 0. , 1. ,
0. , 0. , -0.2637, 1. , 0. , 1. , 0. ,
0. , 1. , 0. , 1. , 0. , 0. , 1. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 1. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , -0.012 , -0.012 , 0. , -0.0028,
-0.0108, 0. , -0.0111, -0.0135, 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , -0.0034,
-0.0027, -0.0725, -0.0673, -0.0625, -0.0582, -0.0065, -0.057 ,
-0.0809, -0.355 ],
[ 1. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 1. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 1. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 1. , 0. , 0. ,
0. , 0. , 1. , 0. , 0. , 0. , 0. ,
0. , 0. , 0.2413, 1. , 0. , 1. , 0. ,
0. , 1. , 0. , 0. , 1. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 1. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , -0.012 , -0.012 , 0. , -0.0028,
-0.0108, 0. , -0.0111, -0.0135, 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 1. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , -0.0034,
-0.0027, -0.0718, -0.0673, -0.0625, -0.0582, -0.0065, -0.057 ,
-0.0809, 0.1579],
[ 1. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 1. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 1. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 1. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 1. ,
0. , 0. , 0.1688, 1. , 0. , 1. , 0. ,
0. , 1. , 0. , 0. , 1. , 0. , 0. ,
0. , 0. , 0. , 1. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 1. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , -0.012 , -0.012 , 0. , -0.0028,
-0.0108, 0. , -0.0111, -0.0135, 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 1. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , -0.0034,
-0.0027, -0.0725, -0.0673, -0.0625, -0.0582, -0.0065, -0.057 ,
-0.0809, 0.1642],
[ 1. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 1. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
1. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 1. , 0. ,
0. , 1. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0.2462, 1. , 0. , 1. , 0. ,
0. , 1. , 0. , 0. , 1. , 1. , 0. ,
0. , 0. , 0. , 1. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 1. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , -0.012 , -0.012 , 0. , -0.0028,
-0.0108, 0. , -0.0111, -0.0135, 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 1. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 1. , 0. , 0. , 1. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0.0004,
-0.0012, -0.069 , -0.0673, -0.0618, -0.0582, -0.0065, -0.057 ,
-0.0809, 0.1713],
[ 1. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 1. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
1. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 1. ,
1. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0.1273, 1. , 0. , 1. , 0. ,
0. , 1. , 0. , 0. , 1. , 0. , 0. ,
0. , 0. , 0. , 1. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 1. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , -0.012 , -0.012 , 0. , -0.0028,
-0.0108, 0. , -0.0111, -0.0135, 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0. ,
0. , 0. , 1. , 0. , 0. , 1. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. , 0.0037,
-0.0023, -0.0633, -0.0673, -0.0625, -0.0582, -0.0065, -0.057 ,
-0.0809, 0.1713]])
Also I have seen one I run SVM, then I get the same Nan, Inf error but it also prints some values as below. Again, there are no NaN anywhere. I have checked it completely. Still I don't know why it is throwing those values.
_unique_labels = _FN_UNIQUE_LABELS.get(label_type, None)
105 if not _unique_labels:
--> 106 raise ValueError("Unknown label type: %r" % ys)
107
108 ys_labels = set(chain.from_iterable(_unique_labels(y) for y in ys))
ValueError: Unknown label type: 117456 0
117457 0
117458 0
117459 0
117460 0
117461 0
117462 0
117463 0
117464 0
117465 0
117466 2
117467 0
117468 0
117469 0
117470 NaN
117471 0
117472 NaN
117473 3
117474 0
117475 NaN
117476 0
117477 NaN
117478 6
117479 0
117480 0
117481 NaN
117482 NaN
117483 0
117484 NaN
1 answers
solution1
0 2015-11-18 03:34:52
There is an additional case you're not checking: the cast to np.float64
It is possible that if you are using a higher precision data type, that the cast down renders a very big number as infinity; when you check the data in its original form, it merrily reports a finite number.
I can't be positive about your exact problem, but here is one way to produce your error: please provide a minimal reproducible example if you want more help. After 8 hours of debugging, I'm sure you have such an example :)
a = np.array([1e100],dtype=np.float128)
# 1e100 still fits within 64 bits. It's not trivial to
# instantiate an array with that large a number without
# doing arithmetic at some point
a *= a
# make it EVEN BIGGER!!! (we're at 1e400 for those keeping track)
a *= a
np.isfinite(a) # prints True: Check
np.isfinite(a.astype(np.float64)) # prints... False. :(
To see if this is a case, just check the data type going into the procedure. It's possible you found such huge value if your were reading from file or doing some arithmetic processing. You're biggest value in 64 bits is something like this but there are plenty of data types to accommodate that value in numpy, as with standard infinite precision integers in python, anything's possible!
In the future, the debugging procedure might be to extract or comment out each conditional before that ValueError and print the data right before hand. You should be able to see exactly where the data goes amok. This process is muddled a little since your using external packages, but you can still get the source and test it this way if you've lost so many hours
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.