运行多个 Tornado 实例会导致 ioloop 已在运行错误

Question

This is how my code looks like, (have excluded some details that aren't really relevant)

from multiprocessing.pool import ThreadPool as Pool

class GetUsers(BaseTask):
    def foo(self):
        pool = Pool()
        try:
            pool.map(self.bar, users)
        finally:
            pool.close()
            pool.join()

    def bar(self, users):
        uuid = users[0]
        ioloopInstance = ioloop.IOLoop().instance()
        isInExperiment = self.isInExperiment(uuid, ioloopInstance)
        log.info(str(uuid)+str(isInExperiment))

    def isInExperiment(self, uuid, ioloop):
        isInExpTag_response =ioloop.run_sync(lambda: self.
                                             fetch_isInExperiment_response(uuid))
        if len(isInExpTag_response.body) > 0:
            return True
        return False

    @gen.coroutine
    def fetch_isInExperiment_response(self, uuid):
        response = yield baz
        raise gen.Return(response)

When I run this, I get ioloop is already running error. I feel that this is because several process running are trying to access the same instance of Tornado and so this error is seen. I have tried reading the documentation of tornado and seen other resources online trying to tackle the same error, but could not find anything helpful.

Can anyone please help me out?

Answer 1

This worked after I changed from multiprocessing.pool import ThreadPool as Pool to from multiprocessing import Pool

I had used to Threadpool as a workaround for fixing Can't pickle <type 'instancemethod'> when using multiprocessing Pool.map() this error, but went on to use copy_reg as defined here https://laszukdawid.com/2017/12/13/multiprocessing-in-python-all-about-pickling/ to resolve the entire issue.