subprocess.Popen的奇怪执行模式

Question

I have a Python script wherein a JAR is called. After the JAR is called, two shell scripts are called. Initially I was doing this:

proc = subprocess.Popen(jar_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.wait()
output, errors = proc.communicate()

proc = subprocess.Popen(prune_command, shell=True)
proc.wait()

proc = subprocess.call(push_command, shell=True)

I have to wait for the first two processes to finish so I use Popen() and the final one I can let it run in the background, so I call() it. I pass shell=True because I want the called shell scripts to have access to environment variables.

The above works, however, I don't get any logging from the JAR process. I've tried calling it this way:

proc = subprocess.call(jar_command)

This logs as I would expect, but the two shell scripts that follow are not executed. Initially I thought the logs just weren't going to stdout but it turns out they're not being executed at all. IE not removing superfluous files or pushing to a database.

Why are the followup shell scripts being ignored?

Answer 1

If you are certain your shell scripts are not running at all, and with the first code everything works - then it must be the java command deadlocks or not terminates correctly using the call() function.

You can validate that by adding a dummy file creation in your bash scripts. Put it in the first line of the script, so if it is executed you'll get the dummy file created. If it's not created, that means the scripts weren't executed, probably due to something with the java execution.

I would have try couple things:

First I would return the Popen instead of call . Instead of using wait() , use communicate() :

Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate . communicate() returns a tuple (stdoutdata, stderrdata) .

proc = subprocess.Popen(jar_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.communicate()

Make sure to check both streams for data (stdout and stderr). You might miss an error the java process raises.

Next I would try disabling the buffer by providing bufsize=0 to Popen . It will eliminate the option it relates to python buffering.

If both options still don't work, try to see if there is an exception by using check_call() :

proc = subprocess.check_call(jar_command)

Run command with arguments. Wait for command to complete. If the return code was zero then return, otherwise raise CalledProcessError .

These options might have the answer; if not, they would help the debugging process. Feel free to comment how this progress.

Answer 2

Most likely, you are forgetting that the processes streams are in fact OS-level buffers with some finite capacity.

For example, if you run a process that produces a lot of output in PIPE mode, and you wait for it to finish before trying to consume whatever that process wrote to output, you have a deadlock:

• The process has filled up the output buffer and is now blocked on writing more data to its output. Until somebody empties the buffer by reading from pipe, the process cannot continue.
• Your program is waiting for the subprocess to finish before you read the data from its buffer.

The correct way is to start a thread in your program that will "drain" the pipe constantly as the process is running and while your main thread is waiting. You must first start the process, then start the drain threads, then wait for process to finish.

For differential diagnosis, check whether the subprocess will run fine with little output (ie as long as the buffer does not fill up, such as a line or two).

The documentation for subprocess has a note about this.