名为 pipe 的客户端返回 ACCESS DENIED 但仅在初始成功连接后

Question

I have a VB.NET process (non-service) that runs as admin and acts as the named pipe server. A DLL is injected into certain processes and the DLL attempts to connect to the server process. This is all on the same host.

I'm getting an ACCESS_DENIED (5) error from the client however, it only occurs after the first two clients successfully connect to the server. So, at least two can successfully connect then all other attempts are denied.

In my test VM, at least 4 instances of the pipe server are created so it doesn't seem to be a simple case of a busy pipe (which I would assume return STATUS_PIPE_BUSY instead of ACCESS_DENIED anyway). I've also added the most permissive pipe security I can (WorldSid/FullControl) and it still fails.

Here is the server pipe code:

Private Sub PipeSetup() 
    For x = 1 To Environment.ProcessorCount  '4 in my test VM 
        t = New Thread(AddressOf PipeSetupProc)
        t.IsBackground = True
        t.Start()
    Next
End Sub

Private Sub PipeSetupProc() 
    Dim s As NamedPipeServerStream = Nothing
    Dim x As PipeSecurity = Nothing

    While True
        Try
            'I've made the access rule the most permissive possible
            'and I'm still getting ACCESS_DENIED from the client.

            x = New PipeSecurity()
            x.AddAccessRule(New PipeAccessRule(New SecurityIdentifier(WellKnownSidType.WorldSid, Nothing), 
                PipeAccessRights.FullControl, Security.AccessControl.AccessControlType.Allow))

            s = New NamedPipeServerStream(MY_PIPE_NAME, PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, 
                PipeTransmissionMode.Message, PipeOptions.Asynchronous, 4096, 4096, x)
            s.WaitForConnection()

            'Do stuff here...

            s.Close()
            s = Nothing
        Catch ex As Exception
           'logerror()
        End Try
    End While
End Sub

During DLLMain(), a connection attempt is made using the following call and this is where it fails with ACCESS_DENIED.

hPipe = CreateFileW(MY_PIPE_NAME, GENERIC_READ | GENERIC_WRITE, 0, NULL, 
    OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_WRITE_THROUGH, 0);

Answer 1

Check your MY_PIPE_NAME on both sides. for c++ it should be similar to "\\.\pipe\MY_PIPE" and on vb side it is just "MY_PIPE".

If the data is always from vb to the dll, you can just declare a function in c++ to be called from vb (inside of an 'extern "c"').

_stdcall int Send2Cpp(char * msg, int length){ /*...*/}

I have a habit of defining my exports in a module.def. If you don't, add the nessasary dll export terms. Otherwise add a file module.def to your vs c++ project, contents

LIBRARY
    EXPORTS
        Send2Cpp

Then from vb, just add

Public Declare Function Send2Cpp Lib "mydll.dll" (byval S as byte, byval L as integer) as integer

You can also supply a callback from vb.

Public Delegate Function GetCppString(ByVal str As IntPtr)
Function LogC(ByVal LI As IntPtr, ByVal lc As Int64) As Int64
        Dim F(lc) As Byte
        Marshal.Copy(LI, F, 0, lc)
        Debug.Print(System.Text.Encoding.ASCII.GetString(F))
End Function
'make sure these vars persist untill dll is done with pointer
'or you will get errors when they are GC'd
dim CBD as GetCppString=AddressOf(GetCppString)
dim CB_PTR as IntPtr = Marshal.GetFunctionPointerForDelegate(CBD)

Then you pas CB_PTR to a dll function, where you can call the function from their. In C++

typedef int(*send2vb)(char *, int);
void RegCB(send2vb f){ /*do something with function pointer*/}

In this case, add RegCB to your module.def and a declare in vb for it. Then you have bidirectional comms between dll and vb. (off course you'd have to wire it all together. In your dllmain, just wait for the function pointer to be non-null before sending data. PS I just mention this as it turns out to be much more efficient than pipes or similar, especially if you use functions that match your use case (in mine, it was video from a c++ dll to vb and back).