如何在进程之间共享 aiohttp.ClientSession()？

Question

I'm trying to create a shared resource in which I can use an aiohttp.ClientSession() object to fetch certain data. This allows me to use parallelization to speed up the actual calculations.

However, when I try to create a manager in which I use aiohttp.ClientSession, an error is raised: TypeError: no default __reduce__ due to non-trivial __cinit__ .

I'm not sure what is going on and perhaps someone would be able to help me out on what this error means and how I could actually initialize a manager that used an aiohttp.ClientSession.

import asyncio
import aiohttp

# import nest_asyncio    

from multiprocessing.managers import BaseManager

class Session:
    def __init__(self, loop):
        asyncio.set_event_loop(loop)
        self.session = aiohttp.ClientSession()

class MyManager(BaseManager):
    pass

# nest_asyncio.apply()

MyManager.register('Session', Session)
loop = asyncio.get_event_loop()

with MyManager() as manager:
    session = manager.Session(loop)

As a fallback solution, I could use requests, which seems to have no problems when registered in a BaseManager. However, this would seriously bottleneck my process, as I have a fairly large workload running over 60+ parallel processes in which each data call takes a fairly long time to complete.

import requests

from multiprocessing.managers import BaseManager

class Session:
    def __init__(self):
        self.session = requests.Session()

class MyManager(BaseManager):
    pass

MyManager.register('Session', Session)

with MyManager() as manager:
    session = manager.Session()

I hope someone can help me out and many thanks in advance!

Answer 1

Thanks Sraw! Based on your suggestion I managed to create a manager without errors! According to the asyncio documentation , a convent and preferred way to create a loop is by simply calling asyncio.run() .

import asyncio
import aiohttp

from multiprocessing.managers import BaseManager

class Session:
    def __init__(self, beta_engine=False):        
        process = self._init_session()
        asyncio.run(process)
        
    async def _init_session(self):
        self.session = aiohttp.ClientSession()
        
class MyManager(BaseManager):
    pass

MyManager.register('Session', Session)

with MyManager() as manager:
    session = manager.Session()