How to run in python subprocess without blocking parent process?

Question

I know the questions is similar to: 1 , 2 but answers for these questions didn't solve my problem.

Workflow in nutshell:

1.Web application sends request to backend. (written in django)

2.Backend starts worker process, to do some job by calling:

run_command = [sys.executable, my_python_worker, flags]
logger.debug('Created os update process "%s" ', run_command)
subprocess.Popen(run_command)

and in return parent process sends http 200 to web application:

logger.debug('Sending response http 200')
return Response(status=status.HTTP_200_OK)

3.Progress of subprocess work is monitored by web application, by using backend api.

The problem:

Worker was implemented as a separate python script. To run worker I used Popen object from python subprocess library. The subprocess has successfully started, I was able observe the progress of it work in second shell console. The execution of parent process in backend is also ongoing. I was able to see the log Sending response http 200 but the response http 200 to web application has never came. However at the moment when subprocess ends (because it has ended it work , or I kill it from shell) the missing http 200 response is immediately received by web application.

The question:

As in the title, how to run subprocess in python, that it will not block sending http response by parent process ?

Answer 1

After reading the documentation of subprocess library I found the flag: close_fds . According to documentation:

If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. (Unix only). Or, on Windows, if close_fds is true then no handles will be inherited by the child process. Note that on Windows, you cannot set close_fds to true and also redirect the standard handles by setting stdin, stdout or stderr.

I changed line in my code from:

subprocess.Popen(run_command)

to:

subprocess.Popen(run_command, close_fds=True)

and it solved the problem. It seems that subprocess has acquired and blocked socket used by parent process.