Python - Using Popen.communicate() does not capture the stdout and stderr of a system call as expected

Question

I have a Python script which executes a system call (the call is actually a java program). I am using Popen.communicate() to attempt to capture the stdout and stderr . However, it only works for some Java applications (of which, ALL the java applications I'm calling do display something to the screen when executed directly in the shell).

The code I am using is as follows:

    # Perform the java call
    try:

        p = subprocess.Popen(args,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)

    except OSError as e:
        error_msg = '  Error reading file while executing system command: {0}'.format(e.message)

    except ValueError as e:
        error_msg = '  Error in arugments while executing system command: {0}'.format(e.message)

    else:
        # Display the results in real time
        for line in iter(p.stdout.readline, ''):
            sys.stdout.write(line)

        # Grab the stdout and stderr from the process.
        output, err = p.communicate()

    finally:
        # Check to see if an internal error occurred. If not, flip the flag
        if not err and not error_msg:
            java_completed = True
        else:
            java_completed = False

In some cases, the line sys.stdout.write(line) is never called, and in other's it is.

Does anyone have any experience with this?

Answer 1

The first thing I notice is that you're reading everything from the process stdout, and then trying to do communicate afterwards: you've already read everything from the pipe, so the output result from p.communicate() will always be empty.

But that doesn't explain why sometimes the sys.stdout.write is not being called.

One thing to keep in mind is that there are multiple ways to display things in a console (ie, in a command window). Printing to the processes stdout stream is only one way, and that's the way that you'll be able to capture with the above code. If I recall from Java days, there is some type of Console class that is sometimes used for displaying to the console, this may not use stdout. Also, advanced console display libraries, like curses for instance, don't generally write to stdout, they interact with the console object directly. I don't know how, or if, you can capture output from these types of processes, but if it is possible it would probably be very system-specific.