Python多处理共享内存：列表列表

Question

Is it possible to share memory for a list of lists using multiprocessing.Array() or something similar? Also, is it possible to share a more complex object such as a scipy.sparse.lil_matrix ? How would I go about doing that? Are there any examples for either? This variable doesn't have to be modified, but it consumes a lot of memory (~1GB per process), and I'm hoping shared memory will cut back the memory usage. Or is that incorrect?

Answer 1

In general, no (and Python is especially bad at copy-on-write multiprocessing, because even reading an object means changing its refcount, and that's a write operation).

You could, however, try to work with memory-mapped files. Dump your dataset in one, open it from multiple processes and reap the benefits of your OS's I/O caching.

What are you doing with your dataset?