How can you know the length of the byte-array to deserialize using protobuf-net?

Question

I am trying to send a C# object from one process to another using a Memory-Mapped File, and I am trying to use either BinaryFormatter or protobuf-net. Neither is working - apparently because I am using by necessity a fixed-length byte-array and protobuf-net needs an array that is exactly the correct length?

Using protobuf-net, I get this exception on Deserialize: "ProtoException: 'Unconsumed data left in the buffer; this suggests corrupt input' at the line: "message1 = Serializer.Deserialize(memoryStream);

Here is my code. At this point I'm just trying a simple example in order to get it to work at a basic level: This is the object that I want to send between programs:

[ProtoContract]
public class IpcMessage
{
    public IpcMessage() { }

    [ProtoMember(1)]
    public string title { get; set; }

    [ProtoMember( 2 )]
    public string content { get; set; }
}

Here is the (simplified - I removed the synchronization) code that sends the IpcMessage object:

static void SampleSend()
{
    // Create the memory-mapped file which allows 'Reading' and 'Writing'
    using (MemoryMappedFile mmf = MemoryMappedFile.CreateOrOpen( "MyMmfName", 1024, MemoryMappedFileAccess.ReadWrite ))
    {
        // Create a view-stream for this process, which allows us to write data from offset 0 to 1024 (whole memory)
        using (MemoryMappedViewStream mmvStream = mmf.CreateViewStream( 0, 1024))
        {
            IpcMessage message1 = new IpcMessage();
            message1.title = "test";
            message1.content = "hello world";
            Serializer.Serialize( mmvStream, message1 );
        }
    }
}

and here is the code in the receiving program (simplified a bit):

// Create the memory mapped file..
using (MemoryMappedFile mmf = MemoryMappedFile.CreateOrOpen( "MyMmfName", 1024, MemoryMappedFileAccess.ReadWrite ))
{
    using (MemoryMappedViewAccessor mmvStream = mmf.CreateViewAccessor( 0, 1024, MemoryMappedFileAccess.Read ))
    {
        byte[] buffer = new byte[1024];
        IpcMessage message1;
        int numberBytesRead = mmvStream.ReadArray<byte>( 0, buffer, 0, 1024 );
        var memoryStream = new MemoryStream(buffer);
        // It is at this next line that I get ProtoException: 'Unconsumed data left in the buffer; this suggests corrupt input'
        message1 = Serializer.Deserialize<IpcMessage>( memoryStream );
    }
}

When I was trying to use BinaryFormatter it complained also. Clearly I am doing something very basically wrong.

Looking thru other questions - I see that most implementations seem to have a byte-array that is allocated to just the right length. Here, I don't know the length ahead of time -- it is just a fixed-length 1024-byte array (I chose this size arbitrarily, just for now). Or perhaps it's something obvious that I'm missing?

This is my first time to use a Memory-Mapped File, or protobuf-net. Any help or advice is appreciated - thank you in advance.

Note: I'm using Visual Studio 2017 Enterprise 15.9.6, and this code is targeting .NET Framework 4.0

Answer 1

Option 1: tell MemoryStream the correct number of bytes to use in the optional constructor overload; this will work for all serializers.

Option 2, specifically for protobuf-net: use ProtoReader ; the Deserialize API accepts a Stream or a ProtoReader ; the latter can be constructed with a notional length and will not over-read

Option 3, again protobuf-net: use the *WithLengthPrefix APIs for serialize and deserialize

Note that any option assumes that your code can robustly determine the length, which should usually be communicated separately in whatever "framing" approach you're using. Option 3 deals with this internally, but in many scenarios your own code still needs to be frame-aware so that you don't consume too much and read I to the next message (unless this is UDP with exactly one frame per packet, but... it isn't, because MemoryMappedFile ). You also need to consider what happens if a payload is larger than the expected buffer size.

The simplest way of "framing" would be to reserve 4 bytes at the start of each logical block; serialize starting at offset 4, and then write the number of bytes written back in at offsets 0-thru-3, using fixed 4-byte layout in your choice of endianness (usually "little"). When reading: read the first 4 bytes to get the length, then consume that- many bytes from the segment. This is essentially what *WithLengthPrefix does internally, except a few different layout options are supported.