Java output from process buider overwritten when using BufferedReader

Question

I'm trying to run an external program in Java and to read the output. The program is a Linux application in C++ that runs a data mining algorithm and prints the patterns found on standard output. I want to be able to read that output from my Java app and to show the patterns using a table. The problem is that the size of the output is quite big (as a test it produces 6.5MB in about 30 seconds). I'm using ProcessBuilder and reading the output using an InputStreamReader buffered using a BufferedReader as you can see in the following code:

      String[] cmd = {"./clogen_periodic", selected, support, "-t 4"};
      Process p = new ProcessBuilder(cmd).start();
      input = new BufferedReader (new InputStreamReader(p.getInputStream()));

      while ((line = input.readLine()) != null) {
         ...
         process line;
         ...
      }

The problem is that the output gets corrupted. When I execute the same program on a console the output is correct but when I use the Java app some lines are merged. More precisely output should be like this

      TMEmulation log_pseduo_allocation (34985) (2 45 76 89 90)
      __divw clock timer (8273) (4 6 67 4 2)

but it is like this

      TMEmulation log_pseduo_allocation (34985) (2__divw 45clock 76timer (89 8273) 904) (6 67 4 2)

Any idea about the possible problem?

Thanks a lot in advance, Patricia

Answer 1

A few possibilities all to do with the called program

1) as @Artefacto says the C++ program output might not be fully buffered so call setvbuf to make it consistant. ie the first output is partially buffered and second is not and so first flushes after the end of the second. In general buffering can differ if called from the command line and from a process.

2) The program is multi-threaded and the output behaves differently when called from java and so the output timing differs.

Basically you need to look at the code for the called program to force logging/output to be all through the same call.

Answer 2

You should be reading stdout and stderr in separate threads to avoid blocking issues. I can't say for sure if that will fix your problem but it should be done anyway to avoid other problems you may hit (your app may deadlock waiting on stdout for example).

Luckily there's a very good example with sample code that walks you through this. http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

The article states (see bottom of page 2) that you should always read from stderr and stdout even if you don't need the output to prevent possible deadlocks.

Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.

Answer 3

Try calling in C++ program, setvbuf with the option _IOLBF . The end of the pipe exposed to the C++ is probably unbuffered, while when you run the programs from the command line with | , it's line buffered.

Answer 4

If you're doing a System.out.print() or what ever for debugging in every iteration currently, then try putting all lines from all iterations into one String and give that a try.

Maybe your output method prints out asynchronously. Therefore your printed output may be corrupted but not the one you got from input stream.

Just an idea...