Deserializing List<T> over networkstream issue

Question

Hi I am trying to send and receive some data between client/server application.

Class

[Serializable]
public class ScanSessie
{
    public string UserId { get; set; }
    public int TotalScanned { get; set; }
    public string Status { get; set; }
    public string DeviceId { get; set; }
}

Serializer and Deserializer extension methods:

public static class SerializerDeserializerExtensions
{
    public static byte[] Serializer(this object _object)
    {
        byte[] bytes;
        using (var _MemoryStream = new MemoryStream())
        {
            IFormatter _BinaryFormatter = new BinaryFormatter();
            _BinaryFormatter.Serialize(_MemoryStream, _object);
            bytes = _MemoryStream.ToArray();
        }
        return bytes;
    }

    public static T Deserializer<T>(this byte[] _byteArray)
    {
        T ReturnValue;

        using (var _MemoryStream = new MemoryStream(_byteArray))
        {
            IFormatter _BinaryFormatter = new BinaryFormatter();
            ReturnValue = (T)_BinaryFormatter.Deserialize(_MemoryStream);
        }
        return ReturnValue;
    }
}

The example data I am trying to send and deserialize it is:

    List<ScanSessie> scannerCl = new List<ScanSessie>();
    scannerCl.Add(new ScanSessie { DeviceId = "0x00456321", UserId = "123456", Status = "scannen ...", TotalScanned = 0 });
    scannerCl.Add(new ScanSessie { DeviceId = "0x00456321", UserId = "123456", Status = "scannen ...", TotalScanned = 0 });
    scannerCl.Add(new ScanSessie { DeviceId = "0x00456321", UserId = "123456", Status = "scannen ...", TotalScanned = 0 });
    scannerCl.Add(new ScanSessie { DeviceId = "0x00456321", UserId = "123456", Status = "scannen ...", TotalScanned = 0 });
    scannerCl.Add(new ScanSessie { DeviceId = "0x00456321", UserId = "123456", Status = "scannen ...", TotalScanned = 0 });
    scannerCl.Add(new ScanSessie { DeviceId = "0x00456321", UserId = "123456", Status = "scannen ...", TotalScanned = 0 });

Sending with the following code:

        NetworkStream broadcastStream = statusSocket.GetStream();

        Byte[] broadcastBytes = SerializerDeserializerExtensions.Serializer(scannerCl);

        broadcastStream.Write(broadcastBytes, 0, broadcastBytes.Length);
        broadcastStream.Flush();

Receiving stream code. ignore the n.DataAvailable loop. I am sending for test purposes very small packets way below the buffer that is transfered in one part.

 using (TcpClient client = new TcpClient())
                {
                    await client.ConnectAsync("10.184.32.39", 8888);

                    ConnectionEstabilished?.Invoke();

                    byte[] message = new byte[1024 * 8];
                    int bytesRead;

                    using (NetworkStream n = client.GetStream())
                    {
                        while (true)
                        {
                            bytesRead = 0;

                            try
                            {
                                if(n.CanRead)
                                {
                                    do
                                    {
                                        bytesRead = await n.ReadAsync(message, 0, message.Length);
                                    }
                                    while (n.DataAvailable);
                                }
                                //bytesRead = await n.ReadAsync(message, 0, 4096);
                                //bytesRead = await n.ReadAsync(message, 0, message.Length);
                            }
                            catch (Exception ex)
                            {
                                // some error hapens here catch it          
                                Debug.WriteLine("DashboardClientConnect " + ex.Message + "\n" + ex.InnerException);
                                break;
                            }
                         }

The receiving code will fire an event with the data as byte[] defined as message. On my processing event I try to deserialize it with:

private void Sc_NewDataReceived(byte[] scanner)
{
    try
    {
        var result = scanner.Deserializer<List<ScanSessie>>();
    }
    catch(Exception ex)
    {
        Debug.WriteLine(ex.InnerException);
    }
}

On the deserialize step it will throw an exception (Exception thrown: 'System.Runtime.Serialization.SerializationException' in mscorlib.dll) Innerexception is null.

When I use the extension methods without sending it over network with some example data the deserializing works flawless.

I've also tried to play around with the receiving buffer sizes. That doesn't seems to help also. The example data size is below 1024*8.

If the send data length is 600 bytes for example. Does the receiving end buffer also needs to be the same size? Normally that should be a problem because it reads in chunks.

After 2 days I give up. Any help would be appreciated. I tried to make the question informative by placing the functioning code snippets.

Answer 1

This code has several problems:

 do
 {
     bytesRead = await n.ReadAsync(message, 0, message.Length);
 }
 while (n.DataAvailable);

First, sender might send message bytes in chunks. First you will read one chunk, then you will read another chunk overwriting first one in buffer (because you use the same buffer for all reads, and you always write to index 0).

Also, you have no idea when message was really received completely. Sender might send one chunk, then there is a delay for whatever reason, at this point your while (n.DataAvailable) loop exists and you are trying to deserialize incomplete message.

On the top of it - when you were lucky to receive full message - the rest of the buffer (assuming message length is smaller that buffer size) is filled with zeroes which anyway prevents successful deserialization.

Also - it's not a good idea to use BinaryFormatter to serialize objects over the network. Instead use something like protobuf.

To fix your networking code - first send length of your serialized message. If you have different message types - also send message type. After sending length (and type if necessary) - send message itself:

NetworkStream broadcastStream = statusSocket.GetStream();
Byte[] broadcastBytes = SerializerDeserializerExtensions.Serializer(scannerCl);
var lengthBytes = BitConverter.GetBytes(broadcastBytes.Length);
if (!BitConverter.IsLittleEndian)
     Array.Reverse(lengthBytes);
broadcastStream.Write(lengthBytes, 0, lengthBytes.Length);
broadcastStream.Write(broadcastBytes, 0, broadcastBytes.Length);
broadcastStream.Flush();

Then on receiving side:

byte[] lengthBuffer = new byte[sizeof(int)];
var bytesRead = await n.ReadAsync(lengthBuffer, 0, lengthBuffer.Length);
// check if bytesRead equals buffer size
if (!BitConverter.IsLittleEndian)
    Array.Reverse(lengthBuffer);
var length = BitConverter.ToInt32(lengthBuffer, 0);
// check if length is not too big, otherwise you will crash process with out of memory
var messageBuffer = new byte[length];
bytesRead = await n.ReadAsync(messageBuffer, 0, messageBuffer.Length);
// check if bytesRead equals buffer size
// now you can deserialize

Note that this code is untested, so take care.

Answer 2

I suggest to use WCF, .NET Remoting or something comparable. They are designed for exactly that purpose.

If you write your own transmission channel, normally you have to embedd the serialized data in a data communication protocol. Typically this contains a message length and other information about the transmitted data. This is required, eg to let the receiver know the data/message size and data type. On the receiver side, normally you have no idea about, how many bytes have to be received and what they are.