如何从在不同线程中运行的多个外部命令管理多个stdout？

Question

I want to run multiple external commands from inside my scala program and be able to handle their output (sent to stdout). Is there a way which will allow me to do it? Output of the programs will contain some logging and probably progress info. I need to parse their output and send it to database server in order to save logs, progress and shutdown processes if they return an error. Important aspect is that I don't want to wait for different processes to finish but to get the output continously.

I am running separate external commands this way:

def execCommand(command: String, bId: Long): Future[(Long, Stream[String])] = {
    Future {
      bId -> command.lineStream
    }(executionContext)
  }

"bId" is just my process id. I can't manage how to get the output in other way than using:

Await.result() 

method, on the result of "execCommand" method, but it's not how I want it to work.

I want my program to handle multiple streams of data in multiple threads and manage this data continously. It doesn't need to be scala, solutions in java will be ok as well.

Answer 1

When you start the process, you also start a thread for reading the output of that process.

If you want to manage that output in a single thread, create a BlockingQueue and send the output to the queue. You'd likely want to track which process the output came from, so use a POJO in the queue.

Example

public class CommandOutput {
    private final int process;
    private final String line;
    public CommandOutput(int process, String line) {
        this.process = process;
        this.line = line;
    }
    public int getProcess() {
        return this.process;
    }
    public String getLine() {
        return this.line;
    }
}

public class CommandOutputStreamHandler extends Thread {
    private final int process;
    private final InputStream stream;
    private final BlockingQueue<CommandOutput> queue;
    public CommandOutputStreamHandler(int process, InputStream stream, BlockingQueue<CommandOutput> queue) {
        this.process = process;
        this.stream = stream;
        this.queue = queue;
    }
    @Override
    public void run() {
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(this.stream))) {
            for (String line; (line = reader.readLine()) != null; ) {
                this.queue.add(new CommandOutput(this.process, line));
                Thread.sleep(200); // just for simulating output over time
            }
        } catch (Exception e) {
            throw new RuntimeException(e); // or simply log the exception
        } finally {
            this.queue.add(new CommandOutput(this.process, null));
        }
    }
}

BlockingQueue<CommandOutput> queue = new LinkedBlockingQueue<>();

final int PROCS = 3;
for (int i = 0; i < PROCS; i++) {
    Process p = new ProcessBuilder("cmd.exe", "/c", "dir", "/b")
            .redirectErrorStream(true)
            .start();
    new CommandOutputStreamHandler(i, p.getInputStream(), queue)
            .start();
}

for (int endMarkers = 0; endMarkers < PROCS; ) {
    CommandOutput co = queue.take();
    if (co.getLine() == null) {
        endMarkers++;
    } else {
        System.out.println(co.getProcess() + ": " + co.getLine());
    }
}

Sample Output

0: .classpath
0: .project
1: .classpath
0: .settings
1: .project
0: lib
1: .settings
0: pom.xml
1: lib
0: src
1: pom.xml
0: target
1: src
2: .classpath
1: target
2: .project
2: .settings
2: lib
2: pom.xml
2: src
2: target