C++ named pipe client won't read more than 4096 bytes

Question

I am trying to implement a Windows named pipe client in C++, that will send RPC requests to a named pipe server written in Go. It all works for short server response lengths. However if the length of the server response exceeds 4096 bytes, the client will not read past 4096 bytes and the response is cut short. I have included a minimal reproducible example of the client and server code below, with most of the error handling removed for brevity. To reproduce the error, change "some large data string" in the server code into a string of ~5000 characters.

I have tried the following without any luck:

• Setting the length of all buffers to a much larger value than 4096.
• Tried using both MESSAGE and BYTE mode in the client and server.
• Checked the http response headers: the response is not chunked.

Any advice would be most appreciated.

C++ client code:

//Minimal implementation of C++ named pipe client. Most error handling removed for brevity. 
//Adapted from https://docs.microsoft.com/en-us/windows/win32/ipc/named-pipe-client

#include <windows.h> 
#include <stdio.h>
#include <conio.h>
#include <tchar.h>

#define BUFSIZE 1048576

int _tmain(int argc, TCHAR *argv[]) 
{ 
    HANDLE hPipe; 
    const char  *lpvMessage="POST / HTTP/1.0\r\nHost: localhost\r\nContent-Length: 33\r\n\r\n{\"method\":\"test\",\"params\":[\"\"]}\r\n\n";
    char   chBuf[BUFSIZE]; 
    BOOL   fSuccess = FALSE; 
    DWORD  cbRead, cbToWrite, cbWritten, dwMode; 
    LPTSTR lpszPipename = TEXT("\\\\.\\pipe\\mypipe.ipc"); 

    // Try to open a named pipe then close it - attempt 1. 
    while (1) 
    { 
        hPipe = CreateFile( 
        lpszPipename,   // pipe name 
        GENERIC_READ |  // read and write access 
        GENERIC_WRITE, 
        0,              // no sharing 
        NULL,           // default security attributes
        OPEN_EXISTING,  // opens existing pipe 
        0,              // default attributes 
        NULL);          // no template file 

    // Break if the pipe handle is valid. 
        if (hPipe != INVALID_HANDLE_VALUE) 
        break; 
        // Exit if an error occurs. 
        _tprintf( TEXT("Could not open pipe. GLE=%d\n"), GetLastError() ); 
        return -1;
    } 
    CloseHandle(hPipe);

    // If successful, open pipe again for use. For some reason, pipe must be opened and closed once (attempt 1) before actually using.  
    hPipe = CreateFile( 
    lpszPipename,   // pipe name 
    GENERIC_READ |  // read and write access 
    GENERIC_WRITE, 
    0,              // no sharing 
    NULL,           // default security attributes
    OPEN_EXISTING,  // opens existing pipe 
    0,              // default attributes 
    NULL);          // no template file 

    // The pipe connected; change to message-read mode. 
    dwMode = PIPE_READMODE_MESSAGE;  //PIPE_READMODE_BYTE doesn't solve the problem either; 
    fSuccess = SetNamedPipeHandleState( 
    hPipe,    // pipe handle 
    &dwMode,  // new pipe mode 
    NULL,     // don't set maximum bytes 
    NULL);    // don't set maximum time
    if ( ! fSuccess) 
    {
    _tprintf( TEXT("SetNamedPipeHandleState failed. GLE=%d\n"), GetLastError() ); 
    return -1;
    }

    // Send a message to the pipe server. 
    cbToWrite = (lstrlen(lpvMessage)+1)*sizeof(char);
    fSuccess = WriteFile( 
    hPipe,                  // pipe handle 
    lpvMessage,             // message 
    cbToWrite,              // message length 
    &cbWritten,             // bytes written 
    NULL);                  // not overlapped 

    do 
    { 
    // Read from the pipe. 
    fSuccess = ReadFile( 
    hPipe,    // pipe handle 
    chBuf,    // buffer to receive reply 
    BUFSIZE*sizeof(char),  // size of buffer 
    &cbRead,  // number of bytes read 
    NULL);    // not overlapped 

    if ( ! fSuccess && GetLastError() != ERROR_MORE_DATA )
    break; 

    printf(chBuf);
    } while ( ! fSuccess);  // repeat loop if ERROR_MORE_DATA 

    printf("\n<End of message, press ENTER to terminate connection and exit>");
    _getch();
    CloseHandle(hPipe); 
    return 0; 
}

Go server code:

//Minimal implementation of Golang named pipe server. Most error handling removed for brevity. 
// +build windows

package main

import (
    "github.com/Microsoft/go-winio"
    "io"
    "io/ioutil"
    "log"
    "net"
    "net/http"
    "os"
)

func main() {
    log.Print("Starting IPC server...")
    StartIPCServer()
}

func HandleDefault(w http.ResponseWriter, req *http.Request) {
    body, _ := ioutil.ReadAll(io.LimitReader(req.Body, 1048576)) 
    defer req.Body.Close()
    log.Printf("Received: '%q'", string(body))
    response:= "some large data string" //If length of response plus http headers >4096 bytes, client will not read past 4096.  
    io.WriteString(w, response)
}

func serve(l net.Listener) error {
    http.HandleFunc("/", HandleDefault)
    return http.Serve(l, nil)
}

func StartIPCServer() {
    var c winio.PipeConfig
    c.SecurityDescriptor = ""
    c.MessageMode = true //changing to false (byte mode) does not solve the problem. 
    c.InputBufferSize = 1048576
    c.OutputBufferSize = 1048576

    path:= `\\.\pipe\mypipe.ipc`
    listener, err := winio.ListenPipe(path, &c)
    log.Print("IPC server running!")
    defer listener.Close()

    err = serve(listener)
    if err != nil {
        log.Fatalf("Serve: %v", err)
        os.Exit(1)
    }
}

Answer 1

The solution I have implemented is as follows:

• Call ReadFile first to get the first 4096 (or fewer) bytes.
• Call PeekNamedPipe to check for additional data in the pipe, and get the number of bytes ( cbReadPeek in my case) for the next call to ReadFile. I have found that PeekNamedPipe is not 100% reliable when called once (it sometimes returns 0 bytes read, even when there is more data in the pipe). I have therefor implemented it as a do {...} while ( fSuccess && cbReadPeek == 0 ) , so it loops until there is either data to read, or fSuccess fails with an error (no more data in the pipe).
• Call ReadFile again if there is more data to read. If not, break the loop: if (GetLastError() == ERROR_BROKEN_PIPE) { break; } if (GetLastError() == ERROR_BROKEN_PIPE) { break; }
• On every ReadFile iteration, the exact number of bytes in cbRead / cbReadPeek is copied from the buffer (chBuf) and appended to a string, otherwise we end up with some of the previously read data when the bytes read drops below 4096. Or I guess the buffer could be flushed instead.

Anyway, it appears to be working fine. I would still really like to know why ReadFile only reads a maximum of 4096 bytes at a time from the pipe. I could not find anything in the MS documentation for the function to explain this behaviour, but maybe I'm missing something obvious. Perhaps this question could be left open for a while to give someone a chance to see it and provide some insight?