When can I call multiprocess.close() or process.join() after early return in flask function?

Question

what i wanted to do When the flask application receives a request to to a specific endpoint (eg /scale-up), Since it takes more than 5 minutes to turn on the EC2 instance, I try to run that logic in a different process and send a response directly(don't wait until that process turn on EC2 instance) to the user who sent the request.

But when this happens, I don't know when I can close() and join() the process.

To recap, When a request comes in, it only 'executes' a process that takes a long time, and apart from that I just send a OK response. After that, how can I close and join this process?

I tried to use flask's teardown_request decorator, but this also doesn't seem to be the right way.

Here is my code.

import multiprocessing
from flask import Flask, Response
from multiprocessing import Pool

app = Flask(__name__)

@app.route(f'/scale-up')
def send_scale_up_request():
    try:
        p = Pool(4)

        p.apply_async(scale_up_instance)

        logger.info("Executing to restart the instance...")
        return Response("Executing to restart the instance...")
        # where can I put p.close() and p.join()??

    except Exception as e:
        logger.error("Error occured")
        logger.error(e)
        logger.error(traceback.format_exc(limit=None))

def scale_up_instance():
    # turn on EC2 instance using aws sdk(boto3)
    .....

Thank you very much. This is my first time asking a question here...

Answer 1

There is no reason to be using a multiprocessing pool to be running a single process and so there is also no need for close and join . But even running a single Process , there is really no place or need to put a call to join for it. The process will run asynchronously and end after your current request has terminated.

Update

To prevent defunct processes because you cannot do a join on the started process, instead run scale_up_instance as a thread. If for some reason threading is not sufficient, the thread can start a process that can be joined:

import multiprocessing
from flask import Flask, Response
from threading import Thread
import time

app = Flask(__name__)

@app.route(f'/scale-up')
def send_scale_up_request():
    try:
        Thread(target=scale_up_instance).start()

        logger.info("Executing to restart the instance...")
        return Response("Executing to restart the instance...")

    except Exception as e:
        logger.error("Error occured")
        logger.error(e)
        logger.error(traceback.format_exc(limit=None))

def scale_up_instance():
    # turn on EC2 instance using aws sdk(boto3)
    import time
    print('starting...')
    time.sleep(10)
    print('ending.')
    # If multiprocessing is required because of GIL restrictions then:
    from multiprocessing import Process
    p = Process(target=do_scale_up) # defined elsewhere
    p.start()
    p.join()