Multiprocessing with Queue.queue in Python for numpy arrays

Question

I am reading a video using OpenCV with multithreading and using queue.queue to store the frames sequentially.

Further, I am trying to make use of multiprocessing for parallel processing of frames from the queue, to further speed up the execution time.

But I don't know how to efficiently pass the frames from the queue to multiple processes.

I tried multiprocessing.queue but its too slow.

Here is my current code for the multiprocessing part:

 with concurrent.futures.ProcessPoolExecutor(max_workers=6) as executor:
    while q.not_empty():       
        frame = q.get()
        futures = executor.submit(func, frame)

This works, but it is slow.

How do I code the multiprocessing part with queue?

Using Windows OS, so can't use fork() .

Answer 1

Placing an array onto a multiprocessing.Queue creates a copy of the array, and allocating new memory is prohibitively slow. To avoid this, you can allocate a buffer in shared memory using multiprocessing.shared_memory .

from multiprocessing import shared_memory

import numpy as np


frame_shape = (720, 1280, 3)
frame_dtype = np.uint8

buffer_size = np.prod(frame_shape) * np.dtype(frame_dtype).itemsize
buffer_name = "frame"

shm = shared_memory.SharedMemory(create=True, size=buffer_size, name=buffer_name)

frame = np.ndarray(shape=frame_shape, dtype=frame_dtype, buffer=shm.buf)

In another process:

from multiprocessing import shared_memory

import numpy as np


frame_shape = (720, 1280, 3)
frame_dtype = np.uint8

buffer_size = np.prod(frame_shape) * np.dtype(frame_dtype).itemsize
buffer_name = "frame"

shm = shared_memory.SharedMemory(size=buffer_size, name=buffer_name)

frame = np.ndarray(shape=frame_shape, dtype=frame_dtype, buffer=shm.buf)
frame[:, :, :] = np.tile(np.arange(3), reps=(*frame_shape[:2], 1))

Back in the original process, the change is reflected:

frame

array([[[0, 1, 2],
        [0, 1, 2],
        [0, 1, 2],
        ...,

When it is time to clean up, in the second process:

shm.close()

And in the original process:

shm.close()
shm.unlink()

If you do not unlink the SharedMemory object and you try to create another object with the same name (even after the process that instantiated it finishes executing), creation will fail until you do unlink it.

If you do not maintain a reference to the SharedMemory object, it can end up being garbage collected before you are done, so make sure to store it in a variable like we did above or a list, etc.

Be aware that you are going to need to synchronize access to the buffer with some kind of mutex like a multiprocessing.Lock or a multiprocessing.Semaphore .