使用嵌套对象的Python多处理

Question

I'm writing an optimization algorithm which uses several different initial conditions to increase the chance of finding the global optimum. I'm trying to make the code run faster by using the multiprocessing library, and running the optimizations on different processes.

This is the way my code is basically working now:

from multiprocessing import Process, Queue
from SupportCostModel.SupportStructure import SupportStructure, SupportType

# Method the processes will execute
def optimizeAlgoritm(optimizeObject, qOut):

    optimizeObject.Optimize()
    qOut.put(optimizeObject)

# Method the main thread will execute
def getOptimumalObject(n):

    for i in range(n):

        # Create a new process with a new nested object that should be optimized
        p = Process(target = optimizeAlgoritm, args = (SupportStructure(SupportType.Monopile), qOut))
        processes.append(p)
        p.deamon = True
        p.start()

# Part the main thread is running        
if __name__ == '__main__':

    qOut = Queue()
    processes = []

    # Run the code on 6 processes
    getOptimumalObject(6)

    for i in range(len(processes)):
        processes[i].join()

    # Get the best optimized object and print the resulting value
    minimum = 1000000000000000000000000.

    while not qOut.empty():

        optimizeObject = qOut.get()

        if optimizeObject.GetTotalMass() < minimum:

            bestObject = optimizeObject
            minumum = optimizeObject.GetTotalMass()

    print(bestObject.GetTotalMass())

This code works as long as I only use 4 processes. If I run more than 4, say 6 as in the example, two processes will get stuck at the end of the code and the code will never stop running as the main thread is still stuck at the processes[i].join() . I think the two processes have a problem in the qOut.put() in the optimizeAlgorithm. When I remove the qOut.put() the code exits giving the error that bestObject doesn't exists, as expected. However, the strange thing is that if I print, for example, the objects minimum after the qOut.put() it will print it, but the process will stay stay alive using 0% of my CPU. This forces the main code to stay alive as well.

I'm quite new to the multiprocessing and a read that OOP and multiprocessing don't always work very well hand in hand. Am I using a wrong approach here? It is kind of frustrating as it almost works, but isn't working for more than 4 processes.

Thanks in advance!

Answer 1

I got it to work using pipes to send my objects!

this is the code I used:

from multiprocessing import Process, Pipe
from SupportCostModel.SupportStructure import SupportStructure, SupportType
import random

# Method the processes will execute
def optimizeAlgoritm(optimizeObject, conn):

    optimizeObject.Optimize()

    # Send the optimized object
    conn.send(optimizeObject)

# Method the main thread will execute
def getOptimumalObject(n):

    connections = []

    for i in range(n):

        # Create a pipe for each of the processes that is started
        parent_conn, child_conn = Pipe()

        # Save the parent connections
        connections.append(parent_conn)

        # Create objects that needs to by optimized using different initial conditions
        if i == 0:
            structure = SupportStructure(SupportType.Monopile)
        else:
            structure = SupportStructure(SupportType.Monopile)
            structure.properties.D_mp = random.randrange(4., 10.)
            structure.properties.Dtrat_tower = random.randrange(90., 120.)
            structure.properties.Dtrat_mud = random.randrange(60., 100.)
            structure.properties.Dtrat_mp = random.randrange(60., 100.)
            structure.UpdateAll()

        # Create a new process with a new nested object that should be optimized
        p = Process(target = optimizeAlgoritm, args = (structure, child_conn))
        processes.append(p)
        p.deamon = True
        p.start()

    # Receive the optimized objects
    for i in range(n):
        optimizedObjects.append(connections[i].recv())

# Part the main thread is running        
if __name__ == '__main__':

    processes = []
    optimizedObjects = []

    # Run the code on 6 processes
    getOptimumalObject(6)

    for i in range(len(processes)):
        processes[i].join()

    # Get the best optimized object and print the resulting value
    minimum = 1000000000000000000000000.

    for i in range(len(optimizedObjects)):

        optimizeObject = optimizedObjects[i]

        if optimizeObject.GetTotalMass() < minimum:

            bestObject = optimizeObject
            minumum = optimizeObject.GetTotalMass()

    print(bestObject.GetTotalMass())