Using eventlet pool inside a for loop gives error after first iteration in python

Question

When I run the code below, I get an error during second iteration.
I wonder if it's because I didn't terminate the first pool before generating a new pool? If so, how do I terminate a pool?
I get the same error when I define the pool outside the for loop.

num_parallel_loop = 6;
collect_result = []
for i in range(n):
    pool = eventlet.GreenPool(size=num_parallel_loop)
    for result in pool.imap(func, dictionary.iteritems()):
         collect_result.append(result)

Error:

Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(self.__args, *self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 325, in _handle_workers
while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
AttributeError: '_MainThread' object has no attribute '_state'

Answer 1

I'm not familiar with eventlet, but my brief look at the documentation makes me wonder whether you have done the patching step.

One of the challenges of writing a library like Eventlet is that the built-in networking libraries don't natively support the sort of cooperative yielding that we need. What we must do instead is patch standard library modules in certain key places so that they do cooperatively yield. We've in the past considered doing this automatically upon importing Eventlet, but have decided against that course of action because it is un-Pythonic to change the behavior of module A simply by importing module B.

Therefore, the application using Eventlet must explicitly green the world for itself, using one or both of the convenient methods provided.

 import eventlet httplib2 = eventlet.import_patched('httplib2') 

 import eventlet eventlet.monkey_patch() 

This bug has the same error you reported, and it seems to be a problem with the patching, so have you done the patching at all?

Update

I tried a small example with eventlet, and I didn't see the problem you describe. Look at this example, and see if it's different from what you are doing. If so, put a complete example in your question.

import eventlet

eventlet.monkey_patch()


def func(item):
    k, s = item
    return k * s

dictionary = {1: 'a',
              2: 'b',
              3: 'c',
              4: 'd',
              5: 'e',
              6: 'f',
              7: 'g',
              8: 'h'}
num_parallel_loop = 6
n = 3
collect_result = []
for i in range(n):
    pool = eventlet.GreenPool(size=num_parallel_loop)
    for result in pool.imap(func, dictionary.iteritems()):
        collect_result.append(result)

print(repr(collect_result))