How to read command-line output of cygwin program in real-time in Delphi?

Question

I need to read lengthy command-line output of originally Linux-based Cygwin program. It works great under cmd.exe , printing new line every few seconds.

When I use this code below, which was discussed many times here on SO, ReadFile function does not return until that program stops. Then all output is provided by ReadFile and printed.

How to make that output read by ReadFile as soon as it is available?

MSDN says that ReadFile doesn't return until CR is reached in ENABLE_LINE_INPUT mode, or buffer full. That progam uses Linux line breaks LF , not Windows CRLF . I used small buffer 32 bytes and disabled ENABLE_LINE_INPUT ( By the way what's the right way of disabling it? ).

Maybe ReadFile doesn't return because of some other issue with Cygwin program itself, not just LF line breaks? But it works fine in Windows cmd.exe , why not in Delphi console application?

const
  CommandExe:string = 'iperf3.exe ';
  CommandLine:string = '-c 192.168.1.11 -u -b 1m -t 8 -p 5001 -l 8k -f m -i 2';
  WorkDir:string = 'D:\PAS\iperf3\win32';// no trailing \
var
  SA: TSecurityAttributes;
  SI: TStartupInfo;
  PI: TProcessInformation;
  StdOutPipeRead, StdOutPipeWrite: THandle;
  WasOK,CreateOk: Boolean;
  Buffer: array[0..255] of AnsiChar;//  31 is Ok
  BytesRead: Cardinal;
  Line:ansistring;

  try// except
  with SA do begin
    nLength := SizeOf(SA);
    bInheritHandle := True;
    lpSecurityDescriptor := nil;
  end;
  CreatePipe(StdOutPipeRead, StdOutPipeWrite, @SA, 0);
  try
    with SI do
    begin
      FillChar(SI, SizeOf(SI), 0);
      cb := SizeOf(SI);
      dwFlags := STARTF_USESHOWWINDOW or STARTF_USESTDHANDLES;
      wShowWindow := SW_HIDE;
      hStdInput := GetStdHandle(STD_INPUT_HANDLE); // don't redirect stdin
      hStdOutput := StdOutPipeWrite;
      hStdError := StdOutPipeWrite;
    end;
    Writeln(WorkDir+'\'+CommandExe+' ' + CommandLine);
    CreateOk := CreateProcess(nil, PChar(WideString(WorkDir+'\'+CommandExe+' ' + CommandLine)),
                              @SA, @SA, True,// nil, nil,
                              CREATE_SUSPENDED or CREATE_NEW_PROCESS_GROUP or NORMAL_PRIORITY_CLASS or CREATE_DEFAULT_ERROR_MODE,// 0,
                              nil,
                              PChar(WideString(WorkDir)), SI, PI);
    CloseHandle(StdOutPipeWrite);// must be closed here otherwise ReadLn further doesn't work
    ResumeThread(PI.hThread);
    if CreateOk then
      try// finally
        repeat
          WasOK := ReadFile(StdOutPipeRead, Buffer, SizeOf(Buffer), BytesRead, nil);
          if BytesRead > 0 then
          begin
            Buffer[BytesRead] := #0;
            Line := Line + Buffer;
            Writeln(Line);
          end;
        until not WasOK or (BytesRead = 0);
        ReadLn;
        WaitForSingleObject(PI.hProcess, INFINITE);
      finally
        CloseHandle(PI.hThread);
        CloseHandle(PI.hProcess);
      end;
  finally
    CloseHandle(StdOutPipeRead);
  end;
  except
    on E: Exception do
      Writeln('Exception '+E.ClassName, ': ', E.Message);
  end;

Also: why do we have to close this handle right after CreateProcess? It is used to read program output:

CloseHandle(StdOutPipeWrite);

If I close it at the end of program, program output is Ok, but ReadLn is never read to stop the program.

HOW TO test all this: In one command window you start iperf3 server and let it listen:

D:\PAS\iperf3\win32>iperf3.exe -s -i 2 -p 5001
-----------------------------------------------------------
Server listening on 5001
-----------------------------------------------------------

In another command window you start the client, which immediately connects to server and start printing output every 2 sec:

D:\PAS\iperf3\win32>iperf3.exe -c 192.168.1.11 -u -b 1m -t 8 -p 5001 -l 8k -f m -i 2
Connecting to host 192.168.1.11, port 5001
[  4] local 192.168.1.11 port 52000 connected to 192.168.1.11 port 5001
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-2.00   sec   240 KBytes  0.98 Mbits/sec  30
[  4]   2.00-4.00   sec   240 KBytes  0.98 Mbits/sec  30
[  4]   4.00-6.00   sec   248 KBytes  1.02 Mbits/sec  31
[  4]   6.00-8.00   sec   240 KBytes  0.98 Mbits/sec  30
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-8.00   sec   968 KBytes  0.99 Mbits/sec  0.074 ms  0/121 (0%)
[  4] Sent 121 datagrams
iperf Done.

Server prints output as well, together with client:

Accepted connection from 192.168.1.11, port 36719
[  5] local 192.168.1.11 port 5001 connected to 192.168.1.11 port 52000
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-2.00   sec   240 KBytes   983 Kbits/sec  0.052 ms  0/30 (0%)
[  5]   2.00-4.00   sec   240 KBytes   983 Kbits/sec  0.072 ms  0/30 (0%)
[  5]   4.00-6.00   sec   248 KBytes  1.02 Mbits/sec  0.077 ms  0/31 (0%)
[  5]   6.00-8.00   sec   240 KBytes   983 Kbits/sec  0.074 ms  0/30 (0%)
[  5]   8.00-8.00   sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/0 (nan%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-8.00   sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/121 (0%)
-----------------------------------------------------------
Server listening on 5001
-----------------------------------------------------------

So iperf3 client works great in the command window. Now let's start "my" code in client mode, while iperf3 server is still listening. Server accepts the connection and start printing output

Accepted connection from 192.168.1.11, port 36879
[  5] local 192.168.1.11 port 5001 connected to 192.168.1.11 port 53069
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-2.00   sec   240 KBytes   983 Kbits/sec  0.033 ms  0/30 (0%)
[  5]   2.00-4.00   sec   240 KBytes   983 Kbits/sec  0.125 ms  0/30 (0%)
[  5]   4.00-6.00   sec   248 KBytes  1.02 Mbits/sec  0.106 ms  0/31 (0%)
[  5]   6.00-8.00   sec   240 KBytes   983 Kbits/sec  0.109 ms  0/30 (0%)
[  5]   8.00-8.00   sec  0.00 Bytes  0.00 bits/sec  0.109 ms  0/0 (nan%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-8.00   sec  0.00 Bytes  0.00 bits/sec  0.109 ms  0/121 (0%)
-----------------------------------------------------------
Server listening on 5001
-----------------------------------------------------------

it means iperf3 client is started inside of 'my' code, but it is not printing anything! Only after client finished, 'my' code prints this output:

Connecting to host 192.168.1.11, port 5001
[  4] local 192.168.1.11 port 53069 connected to 192.168.1.11 port 5001
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-2.00   sec   240 KBytes  0.98 Mbits/sec  30
[  4]   2.00-4.00   sec   240 KBytes  0.98 Mbits/sec  30
[  4]   4.00-6.00   sec   248 KBytes  1.02 Mbits/sec  31
[  4]   6.00-8.00   sec   240 KBytes  0.98 Mbits/sec  30
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-8.00   sec   968 KBytes  0.99 Mbits/sec  0.109 ms  0/121 (0%)
[  4] Sent 121 datagrams
iperf Done.

So, cygwin program output behaves differently, depending if it runs inside command window or Delphi console application. And yes, my output handling code with 'Line' is not perfect, but let's find out how to make ReadFile return in real-time, I'll fix the rest.

Answer 1

How to make that output read by ReadFile as soon as it is available?

The problem is not in the code you provided. It is already reading output in realtime (Although there is another problem with the code that is not related, see below) .

You can try it with the following batch file instead of Cygwin executable:

test.bat:

timeout 5
echo "1"
timeout 5
echo "2"
timeout 5
echo "3"

and the following bash shell file:

test.sh:

sleep 5
echo "1"
sleep 5
echo "2"
sleep 5
echo "3"

It works in realtime and outputs text to console as soon as it is available.

So if the problem is not in the Delphi code, it is related to the Cygwin program. We need more information about your Cygwin program to help you further.

MSDN says that ReadFile doesn't return until CR is reached in ENABLE_LINE_INPUT mode, or buffer full. That progam uses linux line breaks LF, not Windows CR LF. I used small buffer 32 bytes, disabled ENABLE_LINE_INPUT - btw what's the right way of disabling it?

You don't need to disable it.

If you've set the buffer to 32 bytes, then as soon as the buffer is full, ReadFile function should return those 32 bytes, even with UNIX line endings.

Maybe ReadFile doesn't return because of some other issue with cygwin program itself, not just LF line breaks?

This is what I suppose. I don't want to guess the possible reasons, but they are not related to difference of line endings.

Yes, non-Windows line endings can make the command wait for a whole buffer to be filled, but cannot cause the ReadFile to block.

But it works fine in Windows cmd.exe, why not in Delphi console application?

Good question, this is strange. On my side it works both in Delphi and cmd. That is why I suppose the problem is related to Cygwin application.

Also: why do we have to close this handle right after CreateProcess? CloseHandle(StdOutPipeWrite);

This is the writing end of the pipe. We don't need the write handle, because we are not writing to the pipe, we are only reading from it. Your Cygwin application is indirectly writing to that pipe.

---

Also, there are two problems in the code that have to be noted:

• You have a Line variable that is of type string and is not initialized. Initialize that to empty string ( Line := '' ) at the beginning of the routine/program.

• As you have UNIX line ending in Buffer , ReadFile will not return unless the buffer is full, thus containing multiple lines. You need to either change the call to WriteLn routine to Write and ignore line endings, or use a parser that separates the lines.

• Line variable should either be cleared after being written to stdout or should directly receive the value of Buffer, like that:

   ... Buffer[BytesRead] := #0; Line := Buffer; // <- Assign directly to Line, do not concatenate // TODO: Use a parser to separate the multiple lines // in `Line` and output then with `WriteLn` or // ignore line endings altogether and just use `Write` Write(Line); ... 

  Unless you do that the size of Line will increase progressively until it contains the whole output, duplicating.

Answer 2

this is a Summary of solution, thanks to experts who advised here:

Many unix-born programs, which can be launched in Windows with Cygwin package, watch the destination of their output. If stdOut is to a console, output is EOL-buffered. It means that as soon as new line is ready, it is printed, no matter how it is separated: CR or CR+LF. If stdOut is to a pipe or file or something else, output is EOF-buffered, because human is not watching the screen. It means that all multiple lines are printed when program finished (unless we use 'flush', but presumably we don't have the source code). In this case we loose all real-time information.

It's easy to check with this code (with definitions from the very top), put it in right after CreateProcess:

    case GetFileType(SI.hStdInput) of
     FILE_TYPE_UNKNOWN:Lines.Add('Input Unknown') ;
     FILE_TYPE_DISK:Lines.Add('Input from a File') ;
     FILE_TYPE_CHAR:Lines.Add('Input from a Console') ;
     FILE_TYPE_PIPE:Lines.Add('Input from a Pipe') ;
    end;
    case GetFileType(SI.hStdOutput) of
     FILE_TYPE_UNKNOWN:Lines.Add('Output Unknown') ;
     FILE_TYPE_DISK:Lines.Add('Output to a File') ;
     FILE_TYPE_CHAR:Lines.Add('Output to a Console') ;
     FILE_TYPE_PIPE:Lines.Add('Output to a Pipe') ;
   end;

If you set your console I/O like this:

  hStdInput := GetStdHandle(STD_INPUT_HANDLE);
  hStdOutput := GetStdHandle(STD_OUTPUT_HANDLE);
  hStdError := GetStdHandle(STD_OUTPUT_HANDLE);

output will be to the console. If you set it like this:

  hStdInput :=GetStdHandle(STD_INPUT_HANDLE);
  hStdOutput:=StdOutPipeWrite;
  hStdError :=StdOutPipeWrite;

output will be to the pipe. Don't forget to close this end:

 CloseHandle(StdOutPipeWrite);

for reasons, explained by experts above, it works great. Without it the program can't exit.

I prefer to customize the console a bit, to know exact size:

  Rect: TSmallRect;
  Coord: TCoord;
  Rect.Left:=0; Rect.Top:=0; Rect.Right:=80; Rect.Bottom:=30;
  Coord.X:=Rect.Right+1-Rect.Left; Coord.Y:=Rect.Bottom+1-Rect.Top;
  SetConsoleScreenBufferSize(GetStdHandle(STD_OUTPUT_HANDLE),Coord);
  SetConsoleWindowInfo(GetStdHandle(STD_OUTPUT_HANDLE),True,Rect);
//  SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE),FOREGROUND_RED OR BACKGROUND_BLUE);// for maniacs

If it is not a Console application but GUI, the console can be created by

AllocConsole();
SetConsoleTitle('Console TITLE');
ShowWindow(GetConsoleWindow(),SW_SHOW);// or SW_HIDE - it will blink

Still, back to the main problem: how to read the real-time output of the third-party program? If you are lucky, and that program prints out to the attached pipe line by line, as soon as they are ready, you just read them as above with

ReadOk := ReadFile(StdOutPipeRead, Buffer, BufferSize, BytesRead, nil);

If program doesn't cooperate, but waits to the very end to fill up the pipe, you have no choice, but to leave it with the console output, as per above. This way program believes that somebody is watching its output (and you really may watch it with SW_SHOW), and prints line by line. Hopefully not very fast, at least 1 line per second. Because you are not just enjoying output, but quickly grabbing those lines from the console, one by one using this quite boring technique..

You may clear console first, before starting program, if you already worked on it, although it's not necessary for the new console:

 Hcwnd:=GetStdHandle(STD_OUTPUT_HANDLE);
 Coord.X:=0; Coord.Y:=0;
 CharsWritten:=0;
 ClearChar:=#0;
 GetConsoleScreenBufferInfo(Hcwnd,BufInfo);
 ConScreenBufSize := BufInfo.dwSize.X * BufInfo.dwSize.Y;// size of the console screen buffer
 FillConsoleOutputCharacter(Hcwnd,           // Handle to console screen buffer
                            Char(ClearChar), // Character to write to the buffer
                            ConScreenBufSize,// Number of cells to write
                            Coord,           // Coordinates of first cell
                            CharsWritten);   // Receive number of characters written
 ResumeThread(PI.hThread);// if it was started with CREATE_SUSPENDED

Apparently this works:

   BufInfo: _CONSOLE_SCREEN_BUFFER_INFO;
   LineBuf,Line:string;
   SetLength(LineBuf, BufInfo.dwMaximumWindowSize.X);// one horizontal line
   iX:=0; iY:=0;
   repeat
    Coord.X:=0; Coord.Y:=iY;
    ReadOk:=ReadConsoleOutputCharacter(Hcwnd,PChar(LineBuf),BufInfo.dwMaximumWindowSize.X,Coord,CharsRead);
    if ReadOk then begin// ReadOk
       if CharsRead > 0 then Line:=Trim(Copy(LineBuf,1,CharsRead)); else Line:='';

and you are getting into horrible programming of repeated reading of the same line, until it is not blank, checking the next line(s) as well in case that program does WriteLn(''). If those few lines are blank, check

if WaitForSingleObject(PI.hProcess,10) <> WAIT_TIMEOUT then QuitReading:=true;

in case program finished in the middle of console. If output got to the bottom of the console, you repeatedly read that line. If it is the same, check WaitForSingleObject. If it is not, even worse - you have to go back few lines up to find your previous line, to ensure that program didn't spit few lines too quickly so you missed them. Programs love to do that before finish.

There is a lot of messy code inside this skeleton, especially for such a lousy programmer like me:

    if iY < (BufInfo.dwMaximumWindowSize.Y-1-1) then begin// not last line
       if (length(Line)>0) then begin// not blank
                                . . .
                                end// not blank
                           else begin// blank
                                . . .
                                end;// blank
                                                     end// not last line
                                                else begin// last line
       if (length(Line)>0) then begin// not blank
                                . . .
                                end// not blank
                           else begin// blank
                                . . .
                                end;// blank
                                                     end;// last line
    Sleep(200);
   until QuitReading;

But it works! It is amazingly printing to the console the real-time data (if you didn't SW_HIDE it), and at the same time your GUI program prints the same lines grabbed from the console and processes them the way you want. When external program finishes, console disappears, and GUI program holds full results.