Multilabel Classification with Feature Selection (scikit-learn)

Question

I am using scikit-learn to solving a multi-label classification problem with a large number of labels. I followed the ideas from one of the core devs of the project (larsmans). It gives me a runtime warning and then eventually a memory error.

>>> clf = Pipeline([('chi2', SelectKBest(chi2, k=1000)),('svm', LinearSVC())])
>>> 
>>> multiclf = OneVsRestClassifier(clf, n_jobs=-1)
>>> 
>>> multiclf.fit(Xtr, ytr)
/home/rsivapr/scikit-learn/sklearn/feature_selection/univariate_selection.py:157: 
RuntimeWarning: invalid value encountered in divide
  chisq /= f_exp
/home/rsivapr/scikit-learn/sklearn/feature_selection/univariate_selection.py:157: 
RuntimeWarning: invalid value encountered in divide
  chisq /= f_exp
/home/rsivapr/scikit-learn/sklearn/feature_selection/univariate_selection.py:157:             
RuntimeWarning: invalid value encountered in divide
  chisq /= f_exp

And then after multiple such warnings, it fails: The error dump below

Process PoolWorker-21:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/threading.py", line 808, in __bootstrap_inner
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/threading.py", line 761, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 325, in _handle_workers
    pool._maintain_pool()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 229, in _maintain_pool
    self._repopulate_pool()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 222, in _repopulate_pool
    w.start()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

/home/rsivapr/scikit-learn/sklearn/feature_selection/univariate_selection.py:157: RuntimeWarning: invalid value encountered in divide
  chisq /= f_exp
Process PoolWorker-22:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
Process PoolWorker-23:
Process PoolWorker-30:
/home/rsivapr/scikit-learn/sklearn/feature_selection/univariate_selection.py:157: RuntimeWarning: invalid value encountered in divide
  chisq /= f_exp
Process PoolWorker-31:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-28:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-32:
Process PoolWorker-29:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-27:
Process PoolWorker-26:
Traceback (most recent call last):
Process PoolWorker-25:
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self.run()
    self.run()
    self.run()
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    task = get()
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    task = get()
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
    return recv()
    return recv()
MemoryError
MemoryError
MemoryError
    return recv()
MemoryError
    return recv()
    return recv()
MemoryError
MemoryError
    return recv()
MemoryError
    return recv()
MemoryError
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
Process PoolWorker-24:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
UnpicklingError: invalid load key, 'k'.
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
Process PoolWorker-1:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
Process PoolWorker-2:
Traceback (most recent call last):
Process PoolWorker-3:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-4:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-6:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
Process PoolWorker-5:
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
    return recv()
MemoryError
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
Process PoolWorker-7:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
UnpicklingError: unpickling stack underflow
Process PoolWorker-8:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
UnpicklingError: invalid load key, ''.
Process PoolWorker-10:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
UnpicklingError: invalid load key, '�'.
Process PoolWorker-21:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/threading.py", line 808, in __bootstrap_inner
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/threading.py", line 761, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 325, in _handle_workers
    pool._maintain_pool()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 229, in _maintain_pool
    self._repopulate_pool()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 222, in _repopulate_pool
    w.start()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

/home/rsivapr/scikit-learn/sklearn/feature_selection/univariate_selection.py:157: RuntimeWarning: invalid value encountered in divide
  chisq /= f_exp
Process PoolWorker-22:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
Process PoolWorker-23:
Process PoolWorker-30:
/home/rsivapr/scikit-learn/sklearn/feature_selection/univariate_selection.py:157: RuntimeWarning: invalid value encountered in divide
  chisq /= f_exp
Process PoolWorker-31:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-28:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-32:
Process PoolWorker-29:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-27:
Process PoolWorker-26:
Traceback (most recent call last):
Process PoolWorker-25:
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self.run()
    self.run()
    self.run()
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    task = get()
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    task = get()
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
    return recv()
    return recv()
MemoryError
MemoryError
MemoryError
    return recv()
MemoryError
    return recv()
    return recv()
MemoryError
MemoryError
    return recv()
MemoryError
    return recv()
MemoryError
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
Process PoolWorker-24:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
UnpicklingError: invalid load key, 'k'.
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
Process PoolWorker-1:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
Process PoolWorker-2:
Traceback (most recent call last):
Process PoolWorker-3:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-4:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process PoolWorker-6:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
Process PoolWorker-5:
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
    return recv()
MemoryError
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
MemoryError
Process PoolWorker-7:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
UnpicklingError: unpickling stack underflow
Process PoolWorker-8:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
UnpicklingError: invalid load key, ''.
Process PoolWorker-10:
Traceback (most recent call last):
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/home/rsivapr/anaconda/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
UnpicklingError: invalid load key, '�'.

Answer 1

Seing all the MemoryErrors, you've likely run out of memory. Try replacing n_jobs=-1 with n_jobs=1. This controls the amount of processes spawned (and thus the amount of memory used).

If that works you can increase n_jobs it as long as you don't run out of memory, ie n_jobs=2, n_jobs=3 and so on.