如何使用python subprocess.call，将stdout的副本发送到日志文件，同时检测第一个命令的结果

Question

My python script needs to invoke a program, detect if it failed (eg, result != 0 ) and send the output of the program to both stdout like normal plus a log file.

My default shell is bash. I'm using Python 2.7.9

To send output to both stdout and a file I'd normally use tee :

result = subprocess.call('some_program --an-option  | tee -a ' + logfile , shell=True)

However, the pipe in bash will return true even if the first command fails, so this approach fails to detect if the command fails.

If I try to use set -o pipefail in the command (so that the result will indicate if the first command fails) like this:

result = subprocess.call('set -o pipefail && some_program --an_option  | tee -a ' + logfile , shell=True)

I get the error /bin/sh: 1: set: Illegal option -o pipefail

Is there a way in python to invoke a command, send the output to both the normal stdout console and a logfile, and still detect if the command failed?

Note: we have to continue sending some_program's output to stdout since stdout is being sent to a websocket.

Answer 1

I get the error /bin/sh: 1: set: Illegal option -o pipefail

Pass executable='/bin/bash' otherwise /bin/sh is used.

You could implement tee in pure Python:

#!/usr/bin/env python2
import sys
from subprocess import Popen, PIPE

chunk_size = 1 << 13    
p = Popen(["some_program", "--an-option"], stdout=PIPE, bufsize=1)
with p.stdout, open('logfile', 'ab') as logfile:
    for chunk in iter(lambda: p.stdout.read(chunk_size), b''):
        sys.stdout.write(chunk)
        logfile.write(chunk)
if p.wait() != 0:
    raise Error

Answer 2

My preference would to to send stdout to a pipe, and then read the pipe in the Python code. The Python code can write to stdout, a file, etc as required. It would also enable you to set shell=False as setting it to True is a potential security issue, as mentioned in the documentation.

Answer 3

However, the pipe in bash will return true even if the first command fails, so this approach fails to detect if the command fails.

That is not true.
But I think you mean: the 'some_program --an-option | tee -a ' + logfile 'some_program --an-option | tee -a ' + logfile exit status code always is 0 even though fails in any command part.

Well, using multiple commands (when using && or || ) or connecting multiple commands together via pipes causes unreliable exit status code when returned.

Regardless, in the command: some_program --an-option | tee -a ' + logfile some_program --an-option | tee -a ' + logfile logfile is not written if some_program fails. So you don't need to worry regarding exit code.

Anyway the best way to do pipe along with subprocess is creating Popen objects ans handling stdout and stdin :

import subprocess as sp

STATUS_OK = 0
logfile = '/tmp/test.log'
commands = {
    'main' :   'ls /home',
    'pipe_to': 'tee -a ' + logfile
}

process = sp.Popen(commands['main'], shell=True, stdout=sp.PIPE)
# explicitly force waits till command terminate, set and return exit status code
process.wait()

if process.returncode == STATUS_OK:
    stdoutdata = process.communicate()[0]
    # pipe last command output to "tee" command
    sp.Popen(commands['pipe_to'], stdin=sp.PIPE, shell=1).communicate(stdoutdata)
else:
    # do something when command fails 'ls /hom' (in this case) fails
    pass

That is it!
I the last Popen we invoke Popen.communicate() to send the last output from ls command to tee command STDIN.

In the Python doc there's a tiny tutorial called Replacing shell pipeline , maybe you want take a look.