multiprocessing lock doesnt prevent race condition when working with json file

Question

i have a json file that has only one object in it:

incme.json :

{
    "value": 0
}

I am trying to update it using multiprocessing with ProcessPoolExecutor , and prevent a race condition using multiprocessing.Lock :

from concurrent.futures import ProcessPoolExecutor
import multiprocessing as mp
import numpy as np
import json

def inc(tup):   
    lock, ignoreme = tup
    with lock:
        with open('incme.json', 'r') as f:
            data = json.load(f)
    print (data)
    data['value'] += 1
    with lock:
        with open('incme.json', 'w') as f:
            json.dump(data, fp=f, indent=4)
    ignoreme += 1
        
if __name__ == '__main__':
    m = mp.Manager()
    lock = m.Lock()
    
    NUMBER_OF_CPUS = mp.cpu_count()
    # use up to +++ of the cpu`s available 
    USE_UP_TO = 0.5
    
    inc((lock, 1))
    
    with ProcessPoolExecutor(max_workers=np.uint16(NUMBER_OF_CPUS * USE_UP_TO)) as executor:
        for i in range(100):    
            print('inc:')
            executor.submit(inc, ((lock, 1)))

when the code above runs it will make value be 44 or something lower than 101 .

when using the lock in this way:

def inc(tup):   
    lock, ignoreme = tup
    with lock:
        with open('incme.json', 'r') as f:
            data = json.load(f)
        print (data)
       data['value'] += 1
        with open('incme.json', 'w') as f:
            json.dump(data, fp=f, indent=4)
    ignoreme += 1

the value becomes 101 but now it doesn't work asynchronously. what could cause this? does it have something to do with IO related tasks?

Answer 1

Your lock appears to protect too little. Yes, you read atomically, and you write atomically, but you do not do the read-increment-write sequence atomically. There's nothing, eg, to prevent all 100 processes reading up 0, then each of them adding 1 to 0, and then each of them writing out 1 as the new value.

Instead, try removing the second with lock: statement, and indenting the print() and increment statements, so that the entire read-increment-write sequence is done atomically.

Edit

Oops, I see now you already tried that. and already discovered it worked. So I'm just left confused about why you think the original way "should" work; It obviously should not ;-)