如何从Python脚本在后台执行Shell脚本

Question

I am working on executing the shell script from Python and so far it is working fine. But I am stuck on one thing.

In my Unix machine I am executing one command in the background by using & like this. This command will start my app server -

david@machineA:/opt/kml$ /opt/kml/bin/kml_http --config=/opt/kml/config/httpd.conf.dev &

Now I need to execute the same thing from my Python script but as soon as it execute my command it never goes to else block and never prints out execute_steps::Successful , it just hangs over there.

proc = subprocess.Popen("/opt/kml/bin/kml_http --config=/opt/kml/config/httpd.conf.dev &", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, executable='/bin/bash')
if proc.returncode != 0:
    logger.error("execute_steps::Errors while executing the shell script: %s" % stderr)
    sleep(0.05) # delay for 50 ms
else:
    logger.info("execute_steps::Successful: %s" % stdout)

Anything wrong I am doing here? I want to print out execute_steps::Successful after executing the shell script in the background.

All other command works fine but only the command which I am trying to run in background doesn't work fine.

Answer 1

There's a couple things going on here.

First, you're launching a shell in the background, and then telling that shell to run the program in the background. I don't know why you think you need both, but let's ignore that for now. In fact, by adding executable='/bin/bash' on top of shell=True , you're actually trying to run a shell to run a shell to run the program in the background, although that doesn't actually quite work.*

Second, you're using PIPE for the process's output and error, but then not reading them. This can cause the child to deadlock. If you don't want the output, use DEVNULL , not PIPE . If you want the output to process yourself, use proc.communicate() .**, or use a higher-level function like check_output . If you just want it to intermingle with your own output, just leave those arguments off.

_{* If you're using the shell because kml_http is a non-executable script that has to be run by /bin/bash , then don't use shell=True for that, or executable , just make make /bin/bash the first argument in the command line, and /opt/kml/bin/kml_http the second. But this doesn't seem likely; why would you install something non-executable into a bin directory?}

_{** Or you can read it explicitly from proc.stdout and proc.stderr , but that gets more complicated.}

---

At any rate, the whole point of executing something in the background is that it keeps running in the background, and your script keeps running in the foreground. So, you're checking its returncode before it's finished, and then moving on to whatever's next in your code, and never coming back again.

---

It seems like you want to wait for it to be finished. In that case, don't run it in the background—use proc.wait , or just use subprocess.call() instead of creating a Popen object. And don't use & either, of course. While we're at it, don't use the shell, either:

retcode = subprocess.call(["/opt/kml/bin/kml_http",
                           "--config=/opt/kml/config/httpd.conf.dev"],
                          stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if retcode != 0:
    # etc.

Now, you won't get to that if statement until kml_http finishes running.

---

If you want to wait for it to be finished, but at the same time keep doing other stuff, then you're trying to do two things at once in your program, which means you need a thread to do the waiting:

def run_kml_http():
    retcode = subprocess.call(["/opt/kml/bin/kml_http",
                               "--config=/opt/kml/config/httpd.conf.dev"],
                              stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    if retcode != 0:
        # etc.

t = threading.Thread(target=run_kml_http)
t.start()
# Now you can do other stuff in the main thread, and the background thread will
# wait around until kml_http is finished and execute the `if` statement whenever
# that happens

Answer 2

You're using stderr=PIPE, stdout=PIPE which means that rather than letting the stdin and stdout of the child process be forwarded to the current process' standard output and error streams, they are being redirected to a pipe which you must read from in your python process (via proc.stdout and proc.stderr .

To "background" a process, simply omit the usage of PIPE :

#!/usr/bin/python
from subprocess import Popen
from time import sleep

proc = Popen(
    ['/bin/bash', '-c', 'for i in {0..10}; do echo "BASH: $i"; sleep 1; done'])

for x in range(10):
    print "PYTHON: {0}".format(x)
    sleep(1)

proc.wait()

which will show the process being "backgrounded".