Using Multiprocessing with Python Flask 2.7 hangs the main app until process completes

Question

I have set up a Python (2.7) Flask application which uses Flask-SocketIO. I'm running the app somewhat like this

eventlet.monkey_patch(socket=True, select=True, subprocess=True)
socketio = SocketIO(app, cors_allowed_origins='*',
                    async_mode='eventlet')
socketio.run(app, host='0.0.0.0', debug=False,
                 port=5100)

Now, the app runs perfectly fine. I however for 1 API call, need to leverage Python Microprocessing module.

Here is that API method

        jobs = []
        queue = Queue()

        for idx, val in enumerate(myArray):

            process = pool(target=self.getEachNode,
                              args=(val['ip'], self.subscriberId, queue))
            jobs.append(process)


        try:
            print('waiting for someone to inqueue')
            result = queue.get(timeout=120)
            print('finally someone inqueued..................')
            print(result)
        except Exception as ex:
            print('Exception occurred while waiting for queue')

        # terminate all jobs
        for j in jobs:
            j.terminate()

        for j in jobs:
            j.join()

myArray will have like 10 items. So this request will run 10 times.

Whenever that happens, The python console tells me that a new app has been started (as the following code is run again)

if __name__ == "__main__":

    app.wsgi_app = DispatcherMiddleware(
        app.wsgi_app, {getenv("API_BASEURL", "/api"): app})
    socketio.run(app, host='0.0.0.0', debug=False,
                 port=5100, use_reloader=False)

print('python server running on port 5100')

Now what actually happens is, when these processes are running, the main app hangs. I cannot access any other API method. It basically blocks. This also results in socket disconnection because server gets blocked from my understanding.

All I want is for those 10 tasks to execute asynchronously. Should I use Pool for this?

Pool has a method of apply_async , will that help?

Also, I just want 1 response, I dont need all of them to complete (no need to Promise.all).

I cannot use any of Python 3 feature by the way.

Answer 1

I ended up using Celery / Message Queue which allows us to manage async tasks in Flask easily.