使用多处理查找数组中点之间的距离

Question

I've been trying to speed up a practice project using python's multiprocessing library. In the project I have 2 arrays, points and weights, full of x, y coordinates. I'm trying to find the distance between each weight coordinate and point. When I run the program with multiprocessing, the program uses all the computers ram and CPU, and when looking at task manager, up to 20 instances of python are running. I know the program works because it works without multiprocessing but takes around 20 seconds to complete.

Here is the code, and at the bottom is the programming running with Pool.map and Process from the multiprocessing library.

import math
import random
import multiprocessing as mp

screenSize = 1000000

pointsLength = 2000
weightLength = 20000

weightBuffer = screenSize/weightLength
pointBuffer = screenSize/pointsLength

points = []
weights = []
weightPoints = []

counter = 0

for i in range(pointsLength):
    for j in range(pointsLength):
        points.append([random.randint(j * pointBuffer, j * pointBuffer * 2), 
        random.randint(i * pointBuffer, i * pointBuffer * 2)])


for i in range(pointsLength):
    for j in range(pointsLength):
        weightPoints.append([j * weightBuffer, i * weightBuffer])
        weights.append(0)


def FindDistance(i):
    row = math.floor((i / weightLength) / (weightLength / pointsLength))
    col = math.floor((i % weightLength) / (weightLength / pointsLength))
    points1d = (pointsLength * row) + col

    dist = math.dist(points[points1d], weightPoints[i])

    weights[i] = dist

# With Multiprocessing Pool
# sumthing = []
# for i in range(len(weights)):
#     sumthing.append(i)

# with mp.Pool(4) as p:
#     p.map(FindDistance, sumthing)


# With Multiproessing Process
processes = []
for i in range(len(weights)):
    p = mp.Process(target=FindDistance, args=[i])
    p.start()
    processes.append(p)

for process in processes:
    process.join()


# Without Multiprocessing
# for i in range(len(weights)):
#     FindDistance(i)

#     counter += 1

#     if (counter % 25000 == 0):
#         print(counter / 25000)

If anyone knows how I could get multiprocessing to work, where the program would use the 8 cores on my computer without crashing the program because of ram or cpu limitations.

Answer 1

The problem is you're not doing multiprocessing correctly. Specifically your code is missing the if __name__ == '__main__': guard. This is fixed in the code below which uses multiprocessing.Pool (which I think would be the best and easiest way to do what you want). It still takes a number of seconds to execute, but it doesn't overwhelm memory and the CPUs.

Information about needing the if __name__ == '__main__': is buried in the Safe importing of main module subsection of The spawn and forkserver start methods section of the multiprocessing module's documentation.

import math
import random
import multiprocessing as mp

screenSize = 1000000

pointsLength = 2000
weightLength = 20000

weightBuffer = screenSize/weightLength
pointBuffer = screenSize/pointsLength

points = []
weights = []
weightPoints = []

counter = 0

for i in range(pointsLength):
    for j in range(pointsLength):
        points.append([random.randint(j * pointBuffer, j * pointBuffer * 2),
        random.randint(i * pointBuffer, i * pointBuffer * 2)])


for i in range(pointsLength):
    for j in range(pointsLength):
        weightPoints.append([j * weightBuffer, i * weightBuffer])
        weights.append(0)


def FindDistance(i):
    row = math.floor((i / weightLength) / (weightLength / pointsLength))
    col = math.floor((i % weightLength) / (weightLength / pointsLength))
    points1d = (pointsLength * row) + col

    dist = math.dist(points[points1d], weightPoints[i])

    weights[i] = dist


if __name__ == '__main__':  # ADDED

    # With Multiprocessing Pool
    sumthing = []
    for i in range(len(weights)):
        sumthing.append(i)

    with mp.Pool(4) as p:
        p.map(FindDistance, sumthing)

Answer 2

You are iterating over the length of weights(2000 as per your code) and spawning a new process for every iteration, which means 2000 processes. No wonder the CPU and RAM are full.

What you need to do is chunk the weights array into 8 smaller arrays, ideally of equal length. Change the FindDistance function to take an array as a parameter. This parameter will be the smaller chunked array.

def FindDistance(i_arr):
    for i in i_arr:
        row = math.floor((i / weightLength) / (weightLength / pointsLength))
        col = math.floor((i % weightLength) / (weightLength / pointsLength))
        points1d = (pointsLength * row) + col

        dist = math.dist(points[points1d], weightPoints[i])

        weights[i] = dist

def chunking(weights):
    # initialise array of empty arrays with Length equal to number of processors
    smaller_chunks = [ [] i for i in range(8) ]
    for index,item in enumerate(weights):
        index_to_push = index % 8
        smaller_chunks[index_to_push].append(item)

    return smaller_chunks
processes = []
chunks = chunking(weights)
for i in range(len(chunks)):
    p = mp.Process(target=FindDistance, args=[i])
    p.start()
    processes.append(p)

for process in processes:
    process.join()

        

Answer 3

The problem lies in the fact that you are looping over the length of weights . which from your code

for i in range(pointsLength): #pointsLength is 2000
    for j in range(pointsLength):
        weightPoints.append([j * weightBuffer, i * weightBuffer])
        weights.append(0)

is 2000*2000 = 40,000

So you're trying to create 40,000 new processes all at once leading your system to crash. What you can do instead is break down your list of weights into n smaller arrays and use that to just create n new processes

We can split the array using the numpy function numpy.array_split . Now update the FindDistance function to accept the whole new array as an input.

def FindDistance(subarr):
    for i in subarr:
        row = math.floor((i / weightLength) / (weightLength / pointsLength))
        col = math.floor((i % weightLength) / (weightLength / pointsLength))
        points1d = (pointsLength * row) + col

        dist = math.dist(points[points1d], weightPoints[i])

        weights[i] = dist

and finally create n process with the new arguments

subarrays = np.array_split(weights, n)
processes = []
for subarr in subarrays:
    p = mp.Process(target=FindDistance, args=[subarr])
    p.start()
    processes.append(p)

for process in processes:
    process.join()