在python中利用多个处理器共享数据

Question

I have a cpu intensive code which uses a heavy dictionary as data (around 250M data). I have a multicore processor and want to utilize it so that i can run more than one task at a time. The dictionary is mostly read only and may be updated once a day.
How can i write this in python without duplicating the dictionary?
I understand that python threads don't use native threads and will not offer true concurrency. Can i use multiprocessing module without data being serialized between processes?

I come from java world and my requirement would be something like java threads which can share data, run on multiple processors and offers synchronization primitives.

Answer 1

Use shelve for the dictionary. Since writes are infrequent there shouldn't be an issue with sharing it.

Answer 2

You can share read-only data among processes simply with a fork (on Unix; no easy way on Windows), but that won't catch the "once a day change" (you'd need to put an explicit way in place for each process to update its own copy). Native Python structures like dict are just not designed to live at arbitrary addresses in shared memory (you'd have to code a dict variant supporting that in C) so they offer no solace.

You could use Jython (or IronPython) to get a Python implementation with exactly the same multi-threading abilities as Java (or, respectively, C#), including multiple-processor usage by multiple simultaneous threads.

Answer 3

在stdlib中查看一下： http ： //docs.python.org/library/multiprocessing.html有许多很棒的功能可以让你很容易地在进程之间共享数据结构。