How to share memory of a byte array between a C# and Python processes

Question

I have a C# application that will continously allocate memory for data stored in byte arrays. I have another process written in python that will read from these arrays once instantiated. Both processes will be running on a ubuntu machine.

The obvious solution seems to be to share memory between the processes by passing a pointer from the C# process to the python process. However, this has turned out to be difficult.

I've mainly looked at solutions proposed online. Two notable ones are named pipes and mapped memory files. I read the following posts:

Sharing memory between C and Python. Suggested to be done via named pipes: Share memory between C/C++ and Python

The C# application will neither read nor write from the array and the python script will only read from the array. Therefore, this solution doesn't satisfy my efficiency requirements and seems to be a superfluous solution when the data is literally stored in memory.

When i looked at memory mapped files, it seemed as if though that we would allocate memory for these memory files to write the data to. However, the data will already be allocated before the mapped file is used. Thus, it seems inefficient as well.

The second post: https://docs.microsoft.com/en-us/dotnet/standard/io/memory-mapped-files?redirectedfrom=MSDN

The article says: "Starting with the .NET Framework 4, you can use managed code to access memory-mapped files in the same way that native Windows functions access memory-mapped files". Would an ubuntu machine run into potential problems when reading these files in the same way that windows would? And if not, could someone give either a simple example of using these mapped files between the program languages mentioned above as well as pass a reference to these mapped files between the processes, or give a reference to where someone has already done this?

Or if someone knows how to directly pass a pointer to a byte array from C# to python, that would be even better if possible.

Any help is greatly appreciated!

Answer 1

So after coming back to this post four months later, i did not find a solution that satisfied my efficiency needs.

I had tried to find a way to get around having to write a large amount of data, already allocated in memory, to another process. Meaning i would have needed to reallocate that same data taking up double the amount of memory and adding additional overhead even though the data would be read-safe. However, it seemed as though the proper way to solve this, in this case, for two processes running on the same machine would be to use named pipes as they are faster than ie sockets. As Holger stated, this ended up being a question of interprocess-communication.

I ended up writing the whole application in python which happened to be the better alternative in the end anyways, probably saving me a lot of headache.