将带有对象的 function 传递到 concurrent.futures.ProcessPoolExecutor()？

Question

Need help passing objects into the cpu_bound function. The program uses both asyncio and multiprocessing , so if you know both, it would be the best kind of help!

Basically the problem arises at: result = loop.run_in_executor(pool, lambda: cpu_bound(list1, list2, int_var)

I am not able to pass lambda function into the pool, and program errors with: _pickle.PicklingError: Can't pickle <function <lambda> at 0x00000230FDEDD700>: attribute lookup <lambda> on __main__ failed

Here is a mock structure of my program, since the whole program is over 2,000 lines of code:

import ...
# Defining some functions...
.

def cpu_bound(list1, list2, int_var):
  # Some CPU-bound calculations...
  .

async def find_trades(session, list3, list4):
  # Some async function calls
  .
  with concurrent.futures.ProcessPoolExecutor() as pool:
    result = loop.run_in_executor(
        pool, dill.loads(dill.dumps(lambda: cpu_bound(list1, list2, int_var)))
    try:
        await asyncio.wait_for(
            result, timeout=5
        )
    except asyncio.TimeoutError:
        print("Took to long to compute!")

async def run():
  # Some async function calls
  .
  await asyncio.gather(find_trades(session, list3, list4), ...)

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(run())
    loop.close()

Unfortunately, I am relatively new to multiprocessing and might not know a lot of things about restrictions that come with passing objects from the main program's loop into multi-processed parts of it.

Really appreciate all the help!

Answer 1

The following line:

    loop.run_in_executor(
        pool, dill.loads(dill.dumps(lambda: cpu_bound(list1, list2, int_var)))

doesn't seem to make much sense. You are serializing a lambda using dill , but then you're immediately deserializing it back into a lambda before it has a chance to be transferred to the subprocess. This is probably why you are getting an error from pickle, despite attempting to use dill.

But you can avoid a lambda in the first place by passing the intended arguments as positional arguments to run_in_executor :

result = loop.run_in_executor(pool, cpu_bound, list1, list2, int_var)

If the lists contain picklable objects, this should work just fine.