Why does the Java CRLF token does not work with batch file inputs?

Question

background:
I once answered this question that was about flushing two input strings from a Java process to a batch script. Since I found a workaround solution I am still very interested to solve the remaining mystery and find out why the obvious solution is not working.

problem description
See this very simple batch script:

@ECHO OFF
SET /P input1=1st Input: 
SET /P input2=2nd Input: 
ECHO 1st Input: %input1% and 2nd Input: %input2%

If you run this batch script with Java using ProcessBuilder and flush two input strings into it you will notice that only the first input string will be consumed while the second will be ignored. I found out that SET /P command consumes input from pipes when

• CRLF token is found
• by timeout
• by full buffer(1024 Bytes)

My accepted workaround was based on the last two options by using a Thread.sleep(100) statement between the inputs or using a 1024 Byte Buffer for each input.

It always works for single input or in this case the first input because closing the stream has the effect that the batch script reads one input and empty returns all following SET /P statements.

the question
Why is the first option by using the CRLF token "input\\r\\n" not working?

research
I already tried to workaround the String.getBytes() method by creating a Byte Buffer myself using \\x0d and \\x0a as last bytes for CRLF token but it has no effect.

And I tried all other OutputStream wrappers like PrintWriter to check if there is a problem with the flush() implementation without any success.

I created a C++ program that basically does the same as the java programm by using CreateProcess and stangely it works like a charm.

testing code
Not working Java code:

ProcessBuilder builder = new ProcessBuilder("test.bat");
Process process = builder.start();

OutputStream out = process.getOutputStream();
out.write("foo\r\n".getBytes());
out.flush();
out.write("bar\r\n".getBytes());
out.flush();
out.close();

BufferedReader in = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = in.readLine()) != null)
    System.out.println(line);
in.close();

Full working C++ code:

DWORD dwWritten;
char cmdline[] = "test.bat";
CHAR Input1[] = "foo\r\n";
CHAR Input2[] = "bar\r\n";
HANDLE hStdInRd = NULL;
HANDLE hStdInWr = NULL;
SECURITY_ATTRIBUTES saAttr; 
PROCESS_INFORMATION piProcInfo; 
STARTUPINFO siStartInfo;

// Create Pipe 
saAttr.nLength = sizeof(SECURITY_ATTRIBUTES); 
saAttr.bInheritHandle = TRUE; 
saAttr.lpSecurityDescriptor = NULL;
CreatePipe(&hStdInRd, &hStdInWr, &saAttr, 0); 
SetHandleInformation(hStdInWr, HANDLE_FLAG_INHERIT, 0);

// Create Process
ZeroMemory( &piProcInfo, sizeof(PROCESS_INFORMATION));  
ZeroMemory( &siStartInfo, sizeof(STARTUPINFO));
siStartInfo.cb = sizeof(STARTUPINFO); 
siStartInfo.hStdError = GetStdHandle(STD_ERROR_HANDLE);
siStartInfo.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
siStartInfo.hStdInput = hStdInRd;
siStartInfo.dwFlags |= STARTF_USESTDHANDLES;    
CreateProcess(NULL, cmdline, NULL, NULL, TRUE, 0, NULL, NULL, &siStartInfo, &piProcInfo);
CloseHandle(piProcInfo.hProcess);
CloseHandle(piProcInfo.hThread); 

// Write to Pipe
WriteFile(hStdInWr, Input1, (DWORD)strlen(Input1), &dwWritten, NULL);
FlushFileBuffers(hStdInWr);
WriteFile(hStdInWr, Input2, (DWORD)strlen(Input2), &dwWritten, NULL);
FlushFileBuffers(hStdInWr);
CloseHandle(hStdInWr);

the question again
The problem does not make any sense to me and is bugging me a lot. Why does sending the CRLF token from Java does not have any effect on batch file inputs while it does when sending from C++ program?

Answer 1

About "put /p" and the pipe and child processes on Windows OS

Just for a test I've a little expanded your test batch to get four input instead of two Now have a look at this nice test

>type test.txt | test.bat
1st Input:2nd Input:3rd Input:4th Input:1st Input: one   and 2nd Input:  and 3rd
Input:  and 4rd Input:
"--"

>test.bat < test.txt
1st Input:2nd Input:3rd Input:4th Input:1st Input: one   and 2nd Input: two and
3rd Input: three and 4rd Input: four
"--"

The interesting thing in here is that the first example works exactly as the java code (only the first "set /P" receive a value, while the second one works as expected More interesting in if you put a line somewhere in the batch file like this one: wmic Process >> TestProcesses.txt by inspecting the TestProcesses.txt, in my enviromnet, I can see that whith the first method (pipe) is present cmd.exe C:\\Windows\\system32\\cmd.exe /S /D /c" test.bat" that isn't present when we use the second one (redirection)

I we run out new test batch (including wmic diagnostics) from java; when we inspect the TestProcesses we should see two different processes:

java.exe              java  -cp .\build\classes javaappcrlf.JavaAppCRLF
cmd.exe               C:\Windows\system32\cmd.exe /c C:\projects\JavaAppCRLF\test.bat  

as in the first method (pipe) we have a separate process for the batch where "put /p" doesnt works

From the chapter Pipes and CMD.exe of the article Pipes and CMD.exe

This has several side effects: Any newline (CR/LF) characters in the batch_command will be turned into & operators. see StackOverflow If the batch_command includes any caret escape characters ^ they will need to be doubled up so that the escape survives into the new CMD shell.

also the linked article on stack overflow is interesting

About C++ Testing

I make a little change to the c++ program described in Creating a Child Process with Redirected Input and Output just to read a file of four lines ad passing its content to a child process that execute our batch through a pipe and the results are the same of your Java program

Alternative refactoring/workaround

from the findings mentioned above, it comes that a java program that read and write to (temporary) files ( ...I know is not the same thing ) should works; I successfully tested a working solution by changing the builder this way

    ProcessBuilder builder = new ProcessBuilder(
            "cmd",
            "/c",
            "(C:\\projects\\JavaAppCRLF\\test4.bat < C:\\projects\\JavaAppCRLF\\tmp-test4.in)",
            ">",
            "C:\\projects\\JavaAppCRLF\\tmp-test4.out"
    );

Post Scriptum: an interesting note about other shell (ie: bash on ”os x" or linux )

AFAIK not all the other platforms suffers this "issue" in the same way; ie on a bash (os x terminal) I made the following test with a script that acts just as our previous testing under Windows:

cd ~/projects/so-test/java-crlf-token/JavaAppCRLF  
$ cat test.sh
#!/bin/bash - 
# SET /P input1=1st Input: 
echo -n "1st Input:"; 
read input1;
#SET /P input2=2nd Input: 
echo -n "2nd Input:"; 
read input2;
#ECHO 1st Input: %input1% and 2nd Input: %input2%
echo -n "1st Input: ${input1} and 2nd Input: ${input2}"

then the only one changed to the java program is to reference the script:

ProcessBuilder builder = new ProcessBuilder("/Users/userx/projects/so-test/java-crlf-token/JavaAppCRLF/test.sh");

let's see what happens:

$ cat test.txt
abc
cde

# :pipe
$ cat test.txt | test.sh
$ cat test.txt | ./test.sh
1st Input:2nd Input:1st Input: abc and 2nd Input: cde    

# :redirection
$ ./test.sh < test.txt
1st Input:2nd Input:1st Input: abc and 2nd Input: cde

# :java 
$ java -cp build/classes/ javaappcrlf.JavaAppCRLF
1st Input:2nd Input:1st Input: foo
and 2nd Input: bar