多处理中的SystemError

Question

I am using multiprocessing to execute a function for iterative arguments. For too lengthy arrays in the argument I get the following error message:

<multiprocessing.pool.Pool object at 0x545912490>
Exception in thread Thread-2:
Traceback (most recent call last):
  File   "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
put(task)
SystemError: NULL result without error in PyObject_Call

I try to compare a sample of arrays to another sample of arrays. The code is:

import numpy as np
import multiprocessing

def func(args):
    i=args[0]
    array=args[1]
    sample=args[2]
    for j in np.arange(len(sample)):
        temp=0
        for element in array:
            temp+=len(np.where(sample[j]==element)[0])
    if temp==len(array):
        print i,j       #indices of identical arrays in the samples
    return 0

def compare_samples(a,b):
    iter=len(a)
    pool=multiprocessing.Pool()
    iter_args=[]
    for i in range(0,iter):
        iter_args.append([i,a[i],b])
    print pool
    pool.map(func,iter_args)
    return 0

N=100000000       #error if this number is too large
sample1=np.random.random_integers(0,9,size=(N,10))
sample2=np.random.random_integers(0,9,size=(N,10))

compare_samples(sample1,sample2)

I found a similar question ( System error while running subprocesses using Multiprocessing ), but there the solution is only for a special case, I don't see how to apply it generally.

Does anyone know how to fix the error?

Answer 1

Unfortunately, if the answer that you already referenced doesn't work for you, I don't think you're going to find another workaround that allows you to use your current approach. Here is a conversation between a couple of Python devs discussing size limitations of the multiprocessing library, and it doesn't seem like a good solution was reached. It appears that you're butting up against size limitations that don't really have a "fix" without modifying Python itself.

Even if that's changed by now, it (most likely) won't be backported to Python 2.7.

I think you have a few potential solutions:

1. Split up your return data in your worker functions and return them in pieces over a queue . You may need to use a function more like apply_async instead map . This will allow you to control the size of your data you're trying to push between processes at any one time.

2. Use the mmap library and write your results into some shared memory that was set up my the main process.

3. If you want something that's as simple as can be and aren't too worried about speed, you could simply write your results out to a text file, return the file name, and read it back in, in your main process.