将稀疏矩阵转换为密集矩阵时的MemoryError？ （numpy，scikit）

Question

lr = lm.LogisticRegression(penalty='l2', dual=True, tol=0.0001, 
                             C=1, fit_intercept=True, intercept_scaling=1.0, 
                             class_weight=None, random_state=None)

rd = AdaBoostClassifier( base_estimator=lr, 
                                           learning_rate=1, 
                                           n_estimators=20, 
                                           algorithm="SAMME")
##here, i am deleting unnecesseary objects
##print X.shape
##(7395, 412605)
print "20 Fold CV Score: ", np.mean(cross_validation.cross_val_score(rd, X, y, cv=20, scoring='roc_auc'))

When i run this i get this error:

TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.

And then, i changed my code like this:

print "20 Fold CV Score: ", np.mean(cross_validation.cross_val_score(rd, X.toarray(), y, cv=20, scoring='roc_auc'))

Now, i have the following exception:

  File "/usr/lib/python2.7/dist-packages/scipy/sparse/compressed.py", line 559, in toarray
    return self.tocoo(copy=False).toarray(order=order, out=out)
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/coo.py", line 235, in toarray
    B = self._process_toarray_args(order, out)
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/base.py", line 628, in _process_toarray_args
    return np.zeros(self.shape, dtype=self.dtype, order=order)
MemoryError

Any suggestions to solve the issue?

Answer 1

MemoryError means that there isn't enough RAM available on your system to allocate the matrix. Why? Well, a 7395 x 412605 matrix has 3,051,213,975 elements. If they're in the default float64 (usually double in C) datatype, that's 22.7GB. If you convert to lower-precision float32 s (usually float in C), it'd be 11.4GB; maybe that's handle-able on your machine. It'll still be real slow, though.

It seems that AdaBoostClassifier doesn't support sparse inputs (as you can see in the code here ). I don't know offhand if dense representations are necessary for the algorithm or if it's just that the implementation assumed that.