如何在多进程中共享大对象？

Question

I want to do large calculation in python. and I want to use multiprocess to do it. The problem is that I must transport large obj(such as large numpy and list,or list of numpy), from main process to the process created by main process

Here is the problem: 1. if I use like results[i] = pool.apply_async(func, args_tuple) then it will try to pickle the args_tuple ,which cost large time 2. my own class and numpy class is not supported 3. all the object has been prepared in the main process, if I get it in the generated process ,it will cost time also.

How could I solve this problem.. Maybe I should use java if I had predicted this, I looked it up in the doc, but it seems that every solution is the same -- pickle the params from main process to others (cost time)

Answer 1

As far as I know , every way of communicating between process use pickle to transfer the data ( Pipes , Queue and others...)

One way to make your issue work could be to work on shared variables, but you can't work on the same object with multiples process at the same time so it depends on your application. Their is some simple implemented shared variables in multiprocessing : Array and Value which are respectively a 1D array and a single value (int, float, str...). See the documentation here.

But you can also create your own types if you need to, using ctype : see the documentation here.