Errors calling WriteFile from visual basic .net 2010 on a named pipe

Question

I've been banging my head on this one all day, and I just can't make sense of the various (sometimes conflicting) documentation on this. To add to the confusion, at some point during the day, this did (sort of) work - ie didn't throw an access violation error. But the data at the other end of the pipe was nonsense, so I suspect it only 'worked' by accident.

I have a vb.net program (using .net 3.0 so no support for System.IO.NamedPipes :( ) which creates a named pipe and waits for another app to connect and send some data. As an 'ack', I then want the vb program to send back the total length of the message received. I'm able to create the pipe, wait for a connection and receive the message, but it barfs on trying to send the 'ack' using WriteFile . The (current) definition I'm using for WriteFile is based on the corresponding ReadFile , which seems to work fine:

Declare Function ReadFile Lib "kernel32" ( _
    ByVal hFile As Integer, _
    ByRef lpBuffer As Byte, _
    ByVal nNumberOfBytesToRead As Integer, _
    ByRef lpNumberOfBytesRead As Integer, _
    ByVal lpOverlapped As Integer) _
    As Integer

Declare Function WriteFile Lib "kernel32" ( _
    ByVal hFile As Long, _
    ByRef lpBuffer As Byte, _
    ByVal nNumberOfBytesToWrite As Integer, _
    ByRef lpNumberOfBytesWritten As Integer, _
    ByVal lpOverlapped As Integer) _
    As Integer

The stripped down code (error checking and debug printing removed) looks like this:

Dim openMode = PIPE_ACCESS_DUPLEX Or FILE_FLAG_FIRST_PIPE_INSTANCE
Dim pipeMode = PIPE_WAIT Or PIPE_TYPE_MESSAGE Or PIPE_READMODE_MESSAGE
Dim res  ' Result of dll calls

pipeN2Q = CreateNamedPipe("\\.\pipe\N2Q", openMode, pipeMode, 10, 1024, 1024, 2000, IntPtr.Zero)
res = ConnectNamedPipe(pipeN2Q, 0)

Dim rxCount As Integer = 0  ' To hold the number of bytes received
Dim txCount As Integer = 0  ' To hold the number of bytes sent
Dim txReq As Integer = 2  ' To hold the number of bytes we're going to ask to be sent during the 'ack'

Dim dataIn(256) As Byte

res = ReadFile(pipeN2Q, dataIn(0), 256, rxCount, Nothing)

Dim recvBuffer As String = System.Text.Encoding.ASCII.GetString(dataIn, 0, rxCount)

Dim dataOut(2) As Byte
dataOut(0) = 42
dataOut(1) = 43

res = WriteFile(pipeN2Q, dataOut(0), txReq, txCount, Nothing)

Once the code gets to WriteFile , it throws an AccessViolationException - "Attempted to read or write protected memory." I'm assuming this is complaining about the dataOut parameter, but it gives no further details. Things I've tried so far include:

• Changing the declaration of WriteFile so that lpBuffer is declared: ByVal lpBuffer as IntPtr
• Allocating dataOut using Marshal.AllocHGlobal(txReq) and initialised using Marshal.WriteInt16()
• Allocating a large buffer for dataOut (1024 bytes) and initialising them to zeros

To be clear, the message coming from the other app is received perfectly and recvBuffer has the string exactly as sent. I just can't persuade WriteFile to cooperate (except once, perhaps by chance, and I haven't been able to repeat it). I'm hoping it's just something in either the declaration or the initialisation of the variables - any clues?

Answer 1

Based on Hans' suggestion to use pinvoke.net, here is the combination of signatures and calls which eventually worked for me - just in case anyone else can use it. I know I should probably convert those first 3 declarations to follow the same format as ReadFile / WriteFile, but it's working for now...

Declare Function CreateNamedPipe Lib "kernel32" Alias "CreateNamedPipeA" ( _
    ByVal lpName As String, ByVal dwOpenMode As Integer, _
    ByVal dwPipeMode As Integer, ByVal nMaxInstances As Integer, _
    ByVal nOutBufferSize As Integer, ByVal nInBufferSize As Integer, _
    ByVal nDefaultTimeOut As Integer, ByVal lpSecurityAttributes As IntPtr) _
    As SafeFileHandle

Declare Function ConnectNamedPipe Lib "kernel32" ( _
    ByVal hNamedPipe As SafeFileHandle, ByVal lpOverlapped As System.Threading.NativeOverlapped) _
    As SafeFileHandle

Declare Function CloseHandle Lib "kernel32" ( _
    ByVal hFile As SafeFileHandle) _
    As Integer

<DllImport("kernel32.dll", SetlastError:=True)> Friend Shared Function WriteFile( _
ByVal File As SafeFileHandle, _
ByVal Buffer As System.Text.StringBuilder, _
ByVal NumberOfBytesToWrite As Integer, _
ByRef NumberOfBytesWritten As Integer, _
ByRef Overlapped As System.Threading.NativeOverlapped) As <MarshalAs(UnmanagedType.Bool)> Boolean
End Function

<DllImport("kernel32.dll")> Friend Shared Function ReadFile( _
ByVal File As SafeFileHandle, _
ByVal Buffer As System.Text.StringBuilder, _
ByVal NumberOfBytesToRead As Integer, _
ByRef NumberOfBytesRead As Integer, _
ByRef Overlapped As System.Threading.NativeOverlapped) As <MarshalAs(UnmanagedType.Bool)> Boolean
End Function

...

pipeN2Q = CreateNamedPipe("\\.\pipe\N2Q", openMode, pipeMode, 10, 1024, 1024, 2000, IntPtr.Zero)
If Not pipeN2Q.IsInvalid Then
    If Not ConnectNamedPipe(pipeN2Q, Nothing).IsInvalid Then

        Dim rxCount As Integer = 0
        Dim txCount As Integer = 0

        Dim dataIn As New System.Text.StringBuilder

        res = ReadFile(pipeN2Q, dataIn, 256, rxCount, Nothing)
        If res <> 0 Then
            Dim dataOut As New StringBuilder(dataIn.Length.ToString)

            res = WriteFile(pipeN2Q, dataOut, dataOut.Length, txCount, Nothing)
        Else
            Debug.Print("ReadFile ERROR: " & Err.LastDllError)
        End If
    Else
        Debug.Print("ConnectNamedPipe ERROR: " & Err.LastDllError)
    End If
    If CloseHandle(pipeN2Q) <> 0 Then
        Debug.Print("Pipe closed")
    Else
        Debug.Print("CloseHandle ERRPR: " & Err.LastDllError)
    End If
Else
    Debug.Print("CreateNamedPipe ERROR: " & Err.LastDllError)
End If