使用 'spawn' 启动 redis 进程但面临 TypeError: can't pickle _thread.lock objects

Question

I have to use 'spawn' to start process, cause i need to transport cuda tensor between processes. But using 'spawn' to create redis process always facing TypeError: can't pickle _thread.lock objects

for some reason this code delete some part

it seems that only use 'fork' could work fine

import redis
from torch.multiprocessing import Process

class Buffer(Process):

    def __init__(self, name=0, num_peers=2, actor_queue=0, communicate_queue=0):
        Process.__init__(self)
      
        #some arguments
        self.actor_queue = actor_queue
        self.communicate_queue = communicate_queue
       
        pool = redis.ConnectionPool(host='localhost', port=6379, decode_responses=True)
        self.r = redis.Redis(connection_pool=pool)
        self.r.flushall()

    async def write(self, r):
    #do sth

    async def aggregate(self, r):
    #do sth

    def run(self):
        name_process = mp.current_process().name + str(mp.current_process().pid)
        print('starting...', name_process)
        loop = asyncio.get_event_loop()
        asyncio.set_event_loop(loop)
        tasks = asyncio.gather(
            loop.create_task(self.write(self.r)),
            loop.create_task(self.aggregate(self.r)),
        )
        try:
            loop.run_until_complete(tasks)
        finally:
            loop.close()

if __name__ == '__main__':
    mp.set_start_method('spawn')

    queue = mp.Queue(maxsize=5)
    queue.put('sth')
    name = 'yjsp'
    num_peers = 2
    p =Buffer(name, num_peers, queue, c_queue)
    p.start()

Answer 1

problem solved!

we should define pool and other things in run()

Here is the reason: thread live inside process and the process spin up child process to enable parallel. threads need locks to keep resources problems away like multiple process acquire same resources and cause dead lock.

If we define pool in run(), we are already in child process when we get into run() method.

just like this

    def run(self):
        pool = redis.ConnectionPool(host='localhost', port=6379, decode_responses=True)
        r = redis.Redis(connection_pool=pool)
        r.flushall()