Python 在尝试访问共享内存时死亡

Question

I'm working on a piece of code that takes as input some block of shared memory (created using SharedMemory.shared_memory in Python3.8) that contains an ordered list of numbers, as well as a numpy array, with the goal being some final array in shared memory that is the ordered union of these two sets.

The blocks of shared memory are quite large (~8 Gb up to the point where it fails), so to speed up the union the input block of shared memory is divided evenly between some processes, which then individually perform the union on their little chunk, with the result being combined in another (separate) block of shared memory. These unions are performed multiple times, so in practice many blocks of shared memory of increasing size are allocated in the lifetime of the program, however as soon as the union has been completed, the old block of shared memory is unlinked (so at any given time, there are up to two active blocks of shared memory, each with a size ~8Gb at the point of death).

The problem I'm experiencing is that upon a call to access the shared memory block, the process dies without raising any errors. The work is being done in a child process (I'll call it the worker), spawned from another child process that was created from the main process. The worker process doesn't appear to raise any errors, and I've tried using other methods to catch any errors such as using a try:... except Exception as e:... in the worker and then communicating error information back to the main process through a Pipe, but as far as I can tell no errors are being raised and the worker seems to die silently. Specifically, it dies on the following line in the worker process:

shm_block = mp.shared_memory.SharedMemory(name=shm_key)

I've been running the code on a Linux server with 64Gb of RAM, and a call to df –k /dev/shm suggests that I have 32Gb in shared memory. I've been running the program with 8 worker processes (but it fails with fewer processes as well, like 2 and 4 workers), and it seems to run smoothly up to this threshold just above 8Gb where the workers silently die. I've tried to create a minimum reproducible example using the same dataset on a much smaller machine (10Gb RAM) but in this case a MemoryError is raised. I've tried looking into shm_open and mmap which lie underneath the SharedMemory module to see if there is a limit on the size of shared memory block, but I haven't come across anything.

Any advice would be greatly appreciated, and happy to provide any more of the code if that would be helpful. Thanks in advance!

Answer 1

I came accross the same problem. For me, the problem was an Access Violation, that Python does not throw.

from multiprocessing import shared_memory
shm_list = shared_memory.ShareableList([0] * 29, name='shm_xyz')
b = 10 ** 1000
try:
    shm_list[4] = b
except Exception as e:
    print(e)
print('I arrived at that point')

Usually, this should throw an argument out of range error. In my project I am doing pretty much the same, but for some reason (which I was not able to reproduce), it does not throw the exception, but simply crashes instead. When I called the python script from c++ I noticed, that an AccessViolation is thrown. This AccessViolation is thrown by Windows or on a Hardware level as far as I understood, and therefor immediately crashes python.

The solution for me was to avoid using numbers which are too big.

Sorry if I wasn´t of much help, thought I´d still share it in case someone finds this question while searching for the problem like I did.