[英]How to check if a multiprocessing.Lock() is locked?
I'm trying to make a shared resource that can be accessed by multiple processes, but is limited in the number of requests it can handle at the same time.我正在尝试创建一个可以被多个进程访问的共享资源,但它可以同时处理的请求数量受到限制。 In the final product, this shared resource makes a data request based on an API.
在最终产品中,此共享资源基于 API 发出数据请求。
As I'm going to make a lot of request I want to establish multiple separate connections to speed things up a little bit.因为我要提出很多请求,所以我想建立多个单独的连接以加快速度。 In this case I want to establish three simultaneous connections and divide the requests over these connections.
在这种情况下,我想建立三个同时连接并通过这些连接划分请求。
The semaphore seems to work and limits the number of simultaneous function calls to three.信号量似乎工作并将同时 function 调用的数量限制为三个。 However, I don't seem to be able to spread the load of the three calls over the different functions as the multiprocessing.Lock does not support the locked() method like its threading equivalent.
但是,我似乎无法将这三个调用的负载分散到不同的函数上,因为 multiprocessing.Lock 不像它的线程等效项那样支持locked() 方法。
I was wondering if someone could help me to spread these three calls over the different initiated sessions or has a good suggestion on how to approach this problem in a different way?我想知道是否有人可以帮助我将这三个电话分散到不同的启动会话中,或者对如何以不同的方式解决这个问题有一个很好的建议?
Many, many thanks in advance!非常非常感谢提前!
import time
import multiprocessing
import concurrent.futures
from multiprocessing.managers import BaseManager
class Tester:
def __init__(self, sessions=3):
self.semaphore = multiprocessing.BoundedSemaphore(sessions)
self.locka = multiprocessing.Lock()
self.lockb = multiprocessing.Lock()
self.lockc = multiprocessing.Lock()
def call(self, name):
with self.semaphore:
while True:
if not self.locka.locked():
with self.locka:
time.sleep(1)
return self.session_a(name)
if not self.lockb.locked():
with self.lockb:
time.sleep(1)
return self.session_b(name)
if not self.lockc.locked():
with self.lockc:
time.sleep(1)
return self.session_c(name)
def session_a(self, name):
print(f'session_a: {name}')
def session_b(self, name):
print(f'session_b: {name}')
def session_c(self):
print(f'session_c: {name}')
def call_object(obj, name):
obj.call(name)
def main():
BaseManager.register('Tester', Tester)
manager = BaseManager()
manager.start()
inst = manager.Tester()
with concurrent.futures.ProcessPoolExecutor() as executor:
names = ['Alex', 'Brain', 'Carl', 'Derek', 'Eric', 'Frank', 'George', 'Harold']
futures = [executor.submit(call_object, inst, name) for name in names]
if __name__ == "__main__":
main()
Use locks only and:仅使用锁并且:
When trying to acquire a lock don't block so you can try the next one - it will return True if acquired, False if not acquired.尝试获取锁时不要阻塞,因此您可以尝试下一个锁 - 如果获取,它将返回 True,如果未获取,则返回 False。
import time
import multiprocessing
import concurrent.futures
from multiprocessing.managers import BaseManager
class Tester:
def __init__(self, sessions=3):
self.locks = [multiprocessing.Lock(),
multiprocessing.Lock(),
multiprocessing.Lock()]
def call(self, name):
# cycle through locks/sessions till a lock is acquired and
# a result is obtained
# return the result
done = False
while not done:
for lock,session in zip(self.locks,
[self.session_a,
self.session_b,
self.session_c]):
acq = lock.acquire(block=False)
if acq:
#print(f'lock acquired for {session.__name__}:{name}',file=sys.stdout)
try:
time.sleep(1)
result = session(name)
finally:
lock.release()
#print(f'lock for {session.__name__} released')
done = True
#print(result)
break
return result + '!!'
def session_a(self, name):
#print(f'in method session_a: {name}')
return f'session_a: {name}'
def session_b(self, name):
#print(f'in method session_b: {name}')
return f'session_b: {name}'
def session_c(self,name):
#print(f'in method session_c: {name}')
return f'session_c: {name}'
def call_object(obj, name):
return obj.call(name)
def main():
BaseManager.register('Tester', Tester)
manager = BaseManager()
manager.start()
inst = manager.Tester()
with concurrent.futures.ProcessPoolExecutor() as executor:
names = ['Alex', 'Brain', 'Carl', 'Derek', 'Eric', 'Frank', 'George', 'Harold']
futures = [executor.submit(call_object, inst, name) for name in names]
for fut in concurrent.futures.as_completed(futures):
print(f'future result {fut.result()}')
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.