Python Multiprocessing outside of main producing unexpected result

Question

I have a program that makes a call to an API every minute and do some operations, when some condition is met, I want to create a new process that will make calls to another API every seconds and do some operations. Parent process doesn't care the result that this child process produce, the child will run on its own until everything is done. This way the parent process can continue making call to the api every minute and doing operations without interruption.

I looked into multiprocessing. However I cant get it to work outside of main . I tried passing a callback function, but that created unexpected result (where parent process starting running again in parallel at some point).

Another solution I can think of is just create another project, then make a request. However then I will have a lot of repeated code.

What is the best approach to my problem?

example code:

class Main:
    [...]
    foo = Foo()
    child = Child()

    foo.Run(child.Process)
    


class Foo:
    [...]
    def Run(callbackfunction):
        while(True):
            x = self.dataServices.GetDataApi()
            if(x == 1020):
                callbackfunction()
                #start next loop after a minute
    

class Child:
    [...]
    def Compute(self):
        while(True):
            self.dataServics.GetDataApiTwo()
            #do stuff
            #start next loop after a second

    def Process(self):
        self.Compute() # i want this function to run from a new process, so it wont interfer

Edit2: added in multiprocess attempt

class Main:

    def CreateNewProcess(self, callBack):
        if __name__ == '__main__':
            p = Process(target=callBack)
            p.start()
            p.join()
        
    foo = Foo()
    child = Child(CreateNewProcess)

    foo.Run(child.Process)
    


class Foo:
    def Run(callbackfunction):
        while(True):
            x = dataServices.GetDataApi()
            if(x == 1020):
                callbackfunction()
            #start next loop after a minute
    

class Child:
    _CreateNewProcess = None

    def __init__(self, CreateNewProcess):
        self._CreateNewProcess = CreateNewProcess

    def Compute(self, CreateNewProcess):
        while(True):
            dataServics.GetDataApiTwo()
            #do stuff
            #start next loop after a second

    def Process(self):
        self.CreateNewProcess(self.Compute) # i want this function to run from a new process, so it wont interfer

Answer 1

I had to reorganize a few things. Among others:

• The guard if __name__ == '__main__': should include creation of objects and especially calls to functions and methods. Usually it is placed on the global level at the end of code.

• Child objects shouldn't be created in main process. In theory you can do this to use them as containers for necessary data for the child process and then sending them as parameter but I think a separate class should be used for this if seen as necessary. Here I used a simple data parameter which can be anything pickleable.

• It is cleaner to have a function on global level as process target (in my opinion)

Finally it looks like:

from multiprocessing import Process

class Main:
    @staticmethod
    def CreateNewProcess(data):
        p = Process(target=run_child, args=(data,))
        p.start()
        p.join()


class Foo:
    def Run(self, callbackfunction):
        while(True):
            x = dataServices.GetDataApi()
            if(x == 1020):
                callbackfunction(data)
            #start next loop after a minute


class Child:
    def __init__(self, data):
        self._data = data

    def Compute(self):
        while(True):
            dataServics.GetDataApiTwo()
            #do stuff
            #start next loop after a second



# Target for new process. It is cleaner to have a function outside of a
# class for this

def run_child(data):   # "data" represents one or more parameters from
                       # parent to child necessary to run specific child.
                       # "data" must be pickleable.
                       # Can be omitted if unnecessary
    global child
    child = Child(data)
    child.Compute()


if __name__ == '__main__':
    foo = Foo()

    foo.Run(Main.CreateNewProcess)