Python多重处理-通过多个进程修改JSON

Question

I'm trying to modify a JSON file with multiprocessing . I would be able to split the JSON into chunks, so that each process has only access to and modify a certain part of the JSON (so it's guaranteed that no two processes want to modify the same attribute). My question is, how can I share the JSON object between processes so that the changes are reflected on the original object? I know, that multiprocessing passes the object as a copy, so I'd need to use a Manager() , but how exactly can I do that? Currently I have

def parallelUpdateJSON(datachunk):
    for feature in datachunk: 
        #modify chunk

def writeGeoJSON():
    with open('geo.geojson') as f:
        data = json.load(f)
    pool = Pool()
    for i in range(0, mp.cpu_count())):
        #chunk data into a list, so I get listofchunks = [chunk1, chunk2, etc.,]
        #where chunk1 = data[0:chunksize], chunk2 = data[chunksize:2*chunksize] etc.
    pool.map(parallelUpdateJSON, listofchunks)
    pool.close()
    pool.join()
    with open('test_parallel.geojson', 'w') as outfile:
        json.dump(data, outfile)

But of course this passes the chunks as copies, so the original data object doesn't get modified. How can I make it so that data actually gets modified by the processes? Thank you!

Answer 1

It's probably a better idea to avoid synchronous access to your output file. It will be a lot easier to just produce N partial outputs and join them together into a property of your json object. Then you can dump that object to a file.

def process(work):
    return str(work[::-1])

if __name__ == "__main__":
    p = Pool()
    structure = json.loads("""
    { "list":
        [
            "the quick brown fox jumped over the lazy dog",
            "the quick brown dog jumped over the lazy fox"
        ]
    }
    """)
    structure["results"] = p.map(process, structure["list"])
    #print(json.dumps(structure))
    with open("result.json", "w") as f:
        json.dump(structure, f)