[英]synchronization across multiple processes in python
I have a python application that spawns a separate process to do some work (I ran into performance issues using threads due to the GIL (global interpreter lock)). 我有一个python应用程序,它产生一个单独的进程来完成一些工作(由于GIL(全局解释器锁),我遇到了使用线程的性能问题)。 Now what's my methods in python to synchronize shared resources across processes?
现在我在python中使用什么方法来跨进程同步共享资源?
I move data into a queue and a spawn process does the work when as it receives data from that queue. 我将数据移动到一个队列中,当它从该队列接收数据时,一个spawn进程完成工作。 But I need to be able to guarantee that the data is coming out in an orderly fashion, same order as it was copied in so I need to guarantee that only one process at any time can read/write from/to the queue.
但我需要能够保证数据以有序的方式出现,与复制的顺序相同,所以我需要保证在任何时候只有一个进程可以从/向队列读/写。 How do I do that best?
我该怎么做到最好?
Thanks, Ron 谢谢,罗恩
I think you need a Semaphore, check this example code: 我想你需要一个信号量,检查这个示例代码:
import threading
import datetime
class ThreadClass(threading.Thread):
def run(self):
now = datetime.datetime.now()
pool.acquire()
print "%s says hello, World! at time: %s" % (self.getName(),now)
pool.release()
pool = threading.BoundedSemaphore(value=1)
for i in range(10):
t = ThreadClass()
t.start()
Has this output: 有这个输出:
Thread-1 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-2 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-3 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-4 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-5 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-6 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-7 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-8 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-9 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-10 says hello, World! at time: 2013-05-20 18:57:47.609000
Where as: 在哪里:
import threading
import datetime
class ThreadClass(threading.Thread):
def run(self):
now = datetime.datetime.now()
print "%s says hello, World! at time: %s" % (self.getName(),now)
for i in range(10):
t = ThreadClass()
t.start()
Has this output: 有这个输出:
Thread-1 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-2 says hello, World! at time: 2013-05-20 18:58:05.
531000
Thread-4 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-3 says hello, World! at time: 2013-05-20 18:58:05
.531000
Thread-6 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-5 says hello, World! at time: 2013-05-20 18:58:05
.531000
Thread-8 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-7 says hello, World! at time: 2013-05-20 18:58:05
.531000
Thread-10 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-9 says hello, World! at time: 2013-05-20 18:58:0
5.531000
So with a BoundedSemaphore in python you can make sure that before anyone writes to your queue they have to have the semaphore. 因此,使用python中的BoundedSemaphore,您可以确保在任何人写入队列之前,他们必须拥有信号量。 This doesn't ensure your results are added to the queue in the correct order though.
但这并不能确保您的结果以正确的顺序添加到队列中。
EDIT: 编辑:
If you going to do this and keep order your going to need something like this: 如果你要这样做并保持秩序你将需要这样的东西:
import multiprocessing
import datetime
import random
import time
def funfun(number):
time.sleep(random.randint(0,10))
now = datetime.datetime.now()
return "%s says hello, World! at time: %s" % (number,now)
if __name__ == "__main__":
pool = multiprocessing.Pool(10)
for item in pool.imap(funfun,[i for i in range(10)]):
print item
which prints: 打印:
0 says hello, World! at time: 2013-05-21 00:38:48.546000
1 says hello, World! at time: 2013-05-21 00:38:55.562000
2 says hello, World! at time: 2013-05-21 00:38:47.562000
3 says hello, World! at time: 2013-05-21 00:38:51.578000
4 says hello, World! at time: 2013-05-21 00:38:50.578000
5 says hello, World! at time: 2013-05-21 00:38:48.593000
6 says hello, World! at time: 2013-05-21 00:38:52.593000
7 says hello, World! at time: 2013-05-21 00:38:48.593000
8 says hello, World! at time: 2013-05-21 00:38:50.593000
9 says hello, World! at time: 2013-05-21 00:38:51.609000
So with this you could just append to the queue in the right order, and the jobs will wait for their turn to add to the queue. 因此,您可以按正确的顺序附加到队列,并且作业将等待轮到他们添加到队列中。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.