Capturing output from python command as it happens

Question

I have the following python script

#!/usr/bin/env python
import subprocess
import sys
from time import sleep
p = subprocess.Popen(["ls", "-l", "."], stdout=subprocess.PIPE)
output, err = p.communicate()
print "*** Running ls -l command ***\n", output

print "I'm gonna wait 1 second"
sleep(1)

print "Waited..."

sleep(5)

print "Finished"

And the following Java program that executes that script:

protected List<String> runOutputLinesCommand(String scriptsPath) {
    List<String> ret = new ArrayList<String>();

    // constructs the python command to be executed
    String cmd = scriptsPath
            + COMMAND;
    ProcessBuilder pb = new ProcessBuilder(cmd);

    pb.redirectErrorStream(true);
    try {
        // executes the command and waits for its interruption
        Process p = pb.start();
        String s;
        // read from the process's combined stdout & stderr
        BufferedReader stdout = new BufferedReader(new InputStreamReader(
                p.getInputStream()));
        while ((s = stdout.readLine()) != null) {
            // appends the output of the command to the ret variable
            ret.add(s.trim());
        }
        p.waitFor();
        p.getInputStream().close();
        p.getOutputStream().close();
        p.getErrorStream().close();
    } catch (InterruptedException ex) {
        ret.add("script interrupted: " + ex);
    } catch (IOException ex) {
        ret.add("IOException: " + ex);
        ex.printStackTrace(System.out);
    } catch (Exception ex) {
        ret.add("Exception: " + ex);
        ex.printStackTrace(System.out);
    }

    return ret;
}

What I want is the java program print the python line being executed at real time, and not before all the script is executed. I want the Java program to print the output of the python script as it happens. How can I achieve this in java?

Answer 1

You need to print out each line of output from the Python program, instead of (or as well as) appending it to ret :

    while ((s = stdout.readLine()) != null) {
        //ret.add(s.trim());
        System.out.println(s);
    }

Answer 2

As far as my experience goes, in order to be sure the output from you Python script isn't buffered, you also need to disable output buffering, in addition to what DNA suggested. So make sure to call your script with the -u flag to the interpreter; also, sys.stdout.flush() might be necessary.

For more information, see eg Disable output buffering or just google "python output buffering" or "python disable output buffering".