shared memory between processes

Question

I'm playing around with the multiprocessing module in python and trying to parallelize an algorithm that loops through an list with a different increment value each time (modification of the Sieve of Eratosthenes algorithm). Therefore, I want to have a shared list between all of the processes so that all the processes are modifying the same list. I've tried with the multiprocessing.Array function, but when I reach the end of the program the array is still unmodified and still contains all 0's (the value that I initialized it to).

import multiprocessing
import math

num_cores = multiprocessing.cpu_count()

lower = 0
mark = None

def mark_array(k):
    global mark
    index = (-(-lower//k)*k)-lower
    for i in range(index, len(mark), k):
        mark[i] = 1

def sieve(upper_bound, lower_bound):
    size = upper_bound - lower_bound + 1

    global mark
    mark = multiprocessing.Array('i', size, lock=False)
    for i in range(size):
        mark[i] = 0

    klimit = int(math.sqrt(upper_bound)) + 1
    global lower
    lower = lower_bound

    if __name__ == '__main__':
        pool = multiprocessing.Pool(processes=num_cores)
        inputs = list(range(2, klimit+1))
        pool.map(mark_array, inputs)
        pool.close()
        pool.join()

        result = []
        for i in range(size):
            result.append(mark[i])
        print(result)

sieve(200,100)

Pardon the code. It's a bit messy, but I'm just trying to get the shared memory to work before I clean it up.

EDIT: Ok, so I tried the exact same code on a linux machine and there I get my expected output. However, running the same code in VS code on a Windows machine does not. Any idea why?

EDIT#2: This seems to be a Windows specific issue as the Windows OS handles processes differently than Linux. If this is the case, any idea how to solve it?

Answer 1

You could try to use multiprocessing.Manager for your task:

import multiprocessing
import math
from functools import partial

num_cores = multiprocessing.cpu_count()

lower = 0


def mark_array(mark, k):
    index = (-(-lower // k) * k) - lower
    for i in range(index, len(mark), k):
        mark[i] = 1


def sieve(upper_bound, lower_bound):
    size = upper_bound - lower_bound + 1

    klimit = int(math.sqrt(upper_bound)) + 1
    global lower
    lower = lower_bound

    if __name__ == '__main__':
        pool = multiprocessing.Pool(processes=num_cores)
        with multiprocessing.Manager() as manager:
            mark = manager.list(range(size))
            for i in range(size):
                mark[i] = 0

            inputs = list(range(2, klimit + 1))
            foo = partial(mark_array, mark)

            pool.map(foo, inputs)
            pool.close()
            pool.join()

            result = []
            for i in range(size):
                result.append(mark[i])
            print(result)


sieve(200, 100)