C# How should I use NetworkStream more effectively?

Question

Data packets are sent from TcpListener to TcpClient via NetworkStream . Packets aren't big(5 Bytes) but come with great frequency (about 1000 per second or more). Can you give me advice how should I process it most effective? Now I use async for getting stream, which fill buffer, then I cut it to the packets. After that the process is repeated. But at some point I lose true sequence.

s is NetworkStream.

Packet has 2 fields: type(1 Byte (byte)) and value(4 Bytes (int))

MAXVALUEPACKET = 4096

Client Code:

async Task process()
    {
        bool flag = false;
        while (true)
        {
            byte[] actionBuffer;
            flag = false;
            actionBuffer = await ReadFromStreamAsync();
            while (!flag)
            {
                byte type = actionBuffer[0];
                int value = 0;
                if (type > 0)
                {
                    byte[] valueBytes = { actionBuffer[4], actionBuffer[3], actionBuffer[2], actionBuffer[1] };
                    value = BitConverter.ToInt32(valueBytes, 0);
                    actionBuffer = actionBuffer.Skip(5).ToArray();
                    CommonVariables.RawMessages.Add(new KeyValuePair<byte, int>(type, value));
                    OnHandler();
                }
                else
                    flag = true;                     
            }    
        }
    }        
    byte[] buf = new byte[MAXVALUEPACKET];
    async Task<byte[]> ReadFromStreamAsync()
    {
        await s.ReadAsync(buf, 0, MAXVALUEPACKET);
        return buf; 
    }

Answer 1

Setting MAXVALUEPACKET = 5 to read exactly every 5 bytes may help avoid loss of bytes:

        const int MAXVALUEPACKET = 5;
        async Task process()
        {
            while (true)
            {
                var actionBuffer = await ReadFromStreamAsync();
                byte type = actionBuffer[0];
                int value = 0;
                if (type > 0)
                {
                    byte[] valueBytes = { actionBuffer[4], actionBuffer[3], actionBuffer[2], actionBuffer[1] };
                    value = BitConverter.ToInt32(valueBytes, 0);
                    CommonVariables.RawMessages.Add(new KeyValuePair<byte, int>(type, value));
                    OnHandler();
                }
            }
        }

        async Task<byte[]> ReadFromStreamAsync()
        {
            await s.ReadAsync(buf, 0, MAXVALUEPACKET);
            return buf;
        }

The problem of original code logic is when iteration reaches the 820th loop, there is 1 byte left and it makes the logic to read integer value fail. I'm assuming the server always writes exactly 5 bytes in every portions.