Classic problem with executing external commands in Java?

Question

Okay - This one is a bit of a problem for a lot of folks - Since I have yet to see an answer that works, I thought I would express the problem so someone who has figured it out can tell the rest of us.

The problem is that two out of three of the below work just fine -same code.

The instance for reader3 demonstrates the problem. Reader3 cannot read the result of the successful launch of an external file. Attempting to do any type of read (realine, etc.) on either the stdin or stderr InputStream blocks forever:

package Problems;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class RunningProblem {

public static class RunningReader implements Runnable {

    private Process proc;
    private String sName;

    private RunningReader(Process proc1, String sName) {
        this.proc = proc1;
        this.sName = sName;
    }

    public void run() {
        try {                
            // InputStreamReader in = new InputStreamReader(proc.getInputStream());
            // BufferedReader reader = new BufferedReader(in);

            InputStreamReader err = new InputStreamReader(proc.getErrorStream());
            BufferedReader reader = new BufferedReader(err);

            String line = reader.readLine();
            while (line != null) {
                System.out.println(sName + ": " + line);
                line = reader.readLine();
            }
            reader.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

public static void main(String[] args) {
    ExecutorService pool = Executors.newFixedThreadPool(3);
    try {
        Runtime rt = Runtime.getRuntime();

        Process proc1 = rt.exec("ps ax");
        RunningReader reader1 = new RunningReader(proc1, "reader1");

        Process proc2 = rt.exec("ls -l /");
        RunningReader reader2 = new RunningReader(proc2, "reader2");

        Process proc3 = rt.exec("/bin/tar");
        RunningReader reader3 = new RunningReader(proc3, "reader3");

        pool.execute(reader3);
        pool.execute(reader2);
        pool.execute(reader1);

    } catch (Exception ex) {
        System.err.println(ex.getMessage());
    } finally {
        pool.shutdown();
    }
    System.out.println("Launcher.main() Exited.");
}

}

Answer 1

You did not show the output your program yields, but I guess it might be the case, that ps ax and ls -l / do not yield output, but /bin/tar does. The reason is, that the first two commands produce output to stdout but not stderr , whereas the latter will produce output on stderr (since you are not giving valid parameters to tar ) but not on stdout .

Here's the difference when running the commands in the shell:

[axe@gromp tmp]$ ps ax > ps-std.txt 2> ps-err.txt
[axe@gromp tmp]$ ls -l / > ls-std.txt 2> ls-err.txt
[axe@gromp tmp]$ /bin/tar > tar-std.txt 2> tar-err.txt
[axe@gromp tmp]$ ls -lrt
total 18
-rw-r--r-- 1 axe users    0 Mar  5 19:40 ps-err.txt
-rw-r--r-- 1 axe users 7191 Mar  5 19:40 ps-std.txt
-rw-r--r-- 1 axe users    0 Mar  5 19:40 ls-err.txt
-rw-r--r-- 1 axe users  937 Mar  5 19:40 ls-std.txt
-rw-r--r-- 1 axe users    0 Mar  5 19:41 tar-std.txt
-rw-r--r-- 1 axe users  142 Mar  5 19:41 tar-err.txt
[axe@gromp tmp]$ 

Using > to redirect the standard output and 2> to redirect the error output to different files, you can see that tar produces messages on stderr and the other two on stdout (file sizes for the other files are zero, there was no output).

May this be the case? What happens if you run, eg echo "Foo" instead of tar as third process?

Answer 2

I ran your code on my system and it gave me the following output, before exiting normally:

reader3: /bin/tar: You must specify one of the `-Acdtrux' options
reader3: Try `/bin/tar --help' or `/bin/tar --usage' for more information.
Launcher.main() Exited.

There was no output from ps ax nor ls -l / , but running them from the shell confirms that they didn't write anything to standard error. On my system, your code happened to complete normally, but I can imagine situations where this doesn't happen. Do be aware that if a process generates a lot of output on its standard output, a buffer could fill up and that would cause the process to hang.

I'd recommend using a ProcessBuilder instead of Runtime.getRuntime().exec("...") . For one thing, it allows you to redirect the standard error stream into the standard output stream and then not have to worry about which of the two streams to read from.

Answer 3

Facinating (spockian eyebrow) - judging from the above comments, it looks as there are good reasons why so many folks have problems with this - our implementations each seem to work quite differently!

I am running Ubuntu. Interestingly, using ProcessBuilder inverted the problem... but at least none now seem to "block forever" when using it. -At least stderr and stdin were able to be read!

So for me, for now, the rule of thumb seems to be: Use the "old way" (Runtime.getRuntime().exec()) for command-shell ('internal') commands on Ubuntu (Oracle / Sun VM) -Use ProcessBuilder for external commands (like tar, etc.):

Process proc3 = new ProcessBuilder("/bin/tar").start();
RunningReader reader3 = new RunningReader(proc3, "reader3");

...

reader3: /bin/tar: You must specify one of the -Acdtrux' options reader3: Try /bin/tar --help' or `/bin/tar --usage' for more information.

--This is a pretty important operation for many of us ... It would be nice to put together a matrix on what-to-use on which-platform ....? (ie I wonder if OpenJDK would do better on Ubuntu?)

Answer 4

You are probably starting the RunningReaders to late. You start three processes, and probably the first one will sometimes have finished before you start reading the Error-OutputStream in your RunningReader.

The RunningReaders must be started immediately after the process has started.

Under certain conditions even that might not be sufficient. Then you would have to create a wrapper script (for each OS) that captures the output and writes it to a file.