[英]multiprocessing lock doesnt prevent race condition when working with json file
i have a json
file that has only one object in it:我有一个json
文件,其中只有一个 object:
incme.json
: incme.json
:
{
"value": 0
}
I am trying to update it using multiprocessing
with ProcessPoolExecutor
, and prevent a race condition using multiprocessing.Lock
:我正在尝试使用带有ProcessPoolExecutor
的multiprocessing.Lock
处理来更新它,并使用multiprocessing
防止竞争条件:
from concurrent.futures import ProcessPoolExecutor
import multiprocessing as mp
import numpy as np
import json
def inc(tup):
lock, ignoreme = tup
with lock:
with open('incme.json', 'r') as f:
data = json.load(f)
print (data)
data['value'] += 1
with lock:
with open('incme.json', 'w') as f:
json.dump(data, fp=f, indent=4)
ignoreme += 1
if __name__ == '__main__':
m = mp.Manager()
lock = m.Lock()
NUMBER_OF_CPUS = mp.cpu_count()
# use up to +++ of the cpu`s available
USE_UP_TO = 0.5
inc((lock, 1))
with ProcessPoolExecutor(max_workers=np.uint16(NUMBER_OF_CPUS * USE_UP_TO)) as executor:
for i in range(100):
print('inc:')
executor.submit(inc, ((lock, 1)))
when the code above runs it will make value
be 44
or something lower than 101
.当上面的代码运行时,它会使value
为44
或低于101
的值。
when using the lock in this way:以这种方式使用锁时:
def inc(tup):
lock, ignoreme = tup
with lock:
with open('incme.json', 'r') as f:
data = json.load(f)
print (data)
data['value'] += 1
with open('incme.json', 'w') as f:
json.dump(data, fp=f, indent=4)
ignoreme += 1
the value
becomes 101
but now it doesn't work asynchronously.该value
变为101
但现在它不能异步工作。 what could cause this?什么可能导致这个? does it have something to do with IO related tasks?它与 IO 相关任务有关吗?
Your lock appears to protect too little.你的锁似乎保护得太少了。 Yes, you read atomically, and you write atomically, but you do not do the read-increment-write sequence atomically.是的,您以原子方式读取,并且以原子方式写入,但您不会以原子方式执行读取-增量-写入序列。 There's nothing, eg, to prevent all 100 processes reading up 0, then each of them adding 1 to 0, and then each of them writing out 1 as the new value.没有什么可以阻止所有 100 个进程读取 0,然后每个进程都将 1 加到 0,然后每个进程都写出 1 作为新值。
Instead, try removing the second with lock:
statement, and indenting the print()
and increment statements, so that the entire read-increment-write sequence is done atomically.相反,尝试删除第二个with lock:
语句,并缩进print()
和 increment 语句,以便整个读取-递增-写入序列以原子方式完成。
Oops, I see now you already tried that.糟糕,我现在看到你已经尝试过了。 and already discovered it worked.并且已经发现它有效。 So I'm just left confused about why you think the original way "should" work;因此,我只是对您为什么认为原始方式“应该”起作用感到困惑; It obviously should not ;-)它显然不应该;-)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.