Python multiprocessing Pool.imap throws ValueError: list.remove(x): x not in list

Question

I have a very simple test fixture that instantiate and close a test class 'APMSim' in different threads, the class is not picklable, so I have to use multiprocessing Pool.imap to avoid them being transferred between processes:

class APMSimFixture(TestCase):

    def setUp(self):
        self.pool = multiprocessing.Pool()
        self.sims = self.pool.imap(
            apmSimUp,
            range(numCores)
        )

    def tearDown(self):
        self.pool.map(
            simDown,
            self.sims
        )

    def test_empty(self):
        pass

However, when I run the empty Python unittest I encounter the following error:

Error
Traceback (most recent call last):
  File "/home/peng/git/datapassport/spookystuff/mav/pyspookystuff_test/mav/__init__.py", line 87, in tearDown
    self.sims
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value

Why this could happen? Is there a fix to this?

Answer 1

multiprocessing is re-raising an exception from your worker function/child process in the parent process, but it loses the traceback in the transfer from child to parent. Check your worker function, it's that code that's going wrong. It might help to take whatever your worker function is and change:

def apmSimUp(...):
    ... body ...

to:

import traceback

def apmSimUp(...):
    try:
        ... body ...
    except:
        traceback.print_exc()
        raise

This explicitly prints the full, original exception traceback (then lets it propagate normally), so you can see what the real problem is.