Named Pipe in python dies when multiprocessing

Question

first question so please be gentle.

i am using python. when creating a named pipe to a c++ windows program with

    PIPE = open(r'\\.\pipe\NamedPipe','rb+',0)

as global i can read/write from and to the pipe.

    def pipe_writer():
        PIPE.write(some_stuff)

    def pipe_reader():
        data = struct.unpack("byte-type",PIPE.read(number_of_bytes),0) 

    pipe_writer()
    pipe_reader()

this is fine to collect data from the pipe and process the complete data with several functions, one function after the other.

unfortunately i have to process the data bit by bit as i pull it from the pipe with several functions in a serialized manner.

i thought that queueing the data would just do the job so i use the multiprocess module.

when i try to multiprocess i am able to create the pipe and send data once when opening it it after:

    if __name__ == '__main__':
    PIPE = open(r'\\.\pipe\NamedPipe','rb+',0)

    PIPE.write(some_stuff)

when I then try to .start() the functions as processes and read from the pipe I get an error that the pipe doesn't exist or is open in the wrong mode, which can't really be as it works just fine when reading/writing to it without using Process() on the functions AND i can write to it ... even if it's only once.

any suggestions? Also I think I kinda need to use multiprocess as threading doesn't work ... probably ... because of the GIL and slowing stuff down.

Answer 1

If you're in control of the C++ source code too, you can save yourself a lot of code and hassle by moving on to using ZeroMQ or Nanomsg instead of the pipe, and Google Protocol Buffers instead of interpreting a byte stream yourself.

ZeroMQ and Nanomsg are like networks / pipes /IPC on steroids, and are much easier to use than raw pipes, sockets, etc. You have less source code and more functionality : win-win.

Google's protocol Buffers allow you to define data structures (messages) in a language neutral way, and then auto generate source code in C++, Python, Java or whatever. This source code defines structs, classes, etc that represent the messages and also converts them to a standard binary format. That binary data is what you'll send via ZeroMQ. Again, less source code for you to write, more functionality.

This is ideal for getting C++ classes into Python and vice versa.

Answer 2

nanomsg python wrapper is also available on GitHub at Nanomsg Python .

Examples you can see at Examples . I guess this wrapper will serve your purpose. It's always better to use this in place of raw PIPEs. It supports IPC, Between Process and TCP communication patterns.

Moreover it is crossplatform and it's basic implementation is in C. So I guess communication between python and C process can also be made possible.