Multiprocess can not work in flask when I deploy flask to server by using flask + uwsgi + nginx

Question

I'm using flask + uwsgi + nginx to deploy website on server.

In the flask, my code is below, here is what I want to design: every time when I click run model, it would run a model in another process, but the interface would link to the waiting interface immediately.

train_dataset.status = status
db.session.commit()
text = 'Start training your models, please wait for a while or check results several hours later'

# would run a model in another process
task = Process(target=start_train, args=(app.config['UPLOAD_FOLDER'], current_user.id, p.id, p.task), name="training.exe")
task.start()
print(task.pid, task.name)
session[f"{project_name}_train"] = task.pid

# in the meanwhile, link to waiting interface
return render_template('p_computeview.html', project_name=project_name, text=text,
                               status=status, p=p, email=email, not_identify=not_identify)

And when I test in local development environment

app.run()

it's ok, when I click run model, the interface just link to wait interface and I can see the model running logs. But when I deploy to server, I chose uwsgi + nginx + flask.

In uwsgi.ini, I already specify the processes

processes=2
threads=5

But when I click run model, the interface was still, didn't link to waiting interface, however I can see the model running logs, and when finished modeling, the interface would link to waiting interface (which prove that the Process function was not working ??)

my server have 2 cpus, so I think it can support multi process

Can someone help me ? I guess there are some problems in uwsgi or nginx ?

Answer 1

The desire to run separate threads or processes within a request context is a common one. For various reasons, except in very narrow circumstances, it is a desire that leads to frustration. In this case, as soon as task goes out of scope, the Process machinery gets torn down.

If you want to start a long-running task from a request handler, use a framework like celery or Rq , which arrange to run jobs entirely out of process from the http server.