"从 USB 串行端口读取时，SerialPort.BaseStream.ReadAsync 丢弃或加扰字节"

Question

Edit: I've added the sending code and an example of the received output I'm getting.

---

I am reading data from a USB "virtual" serial port connected to an embedded system. I have written two methods for receiving the data, one synchronous and one asynchronous. The synchronous one works, and the asynchronous one loses or scrambles a little bit of the incoming data. I cannot tell why the second one fails.

The method that works calls SerialPort.Read with a read timeout set to zero, and it requests everything in the receive buffer. I check the return value to see how many bytes were actually read and then put the data into a circular buffer for use elsewhere. This method is called by a timer interrupt, and it works perfectly to receive serial data (typically at rates above 1.6 Mbps with no data loss). However, the polling timer has become a problem for me and I would prefer to receive the data asynchronously wrt the rest of my code.

The method that loses data awaits ReadAsync on the serial port BaseStream and loops until cancelled. This approach sort of works, but it often returns the leading byte of a packet out of order, loses a single byte fairly frequently (approximately once every few thousand data bytes), and occasionally loses hundreds of sequential bytes from a packet.

It is possible that there are two completely different problems here, because the loss of larger chunks of data loss seem to be correlated with higher data rates and heavier system activity. That particular part of the problem could potentially be due to buffer overruns -- perhaps through a failure of USB handshaking when the USB scheduler encounters a delay -- but the example I am showing here has only a very small amount of data being transferred at 50 msec intervals, and the system is idle except for this test routine.

I have observed that ReadAsync frequently returns the first byte of a packet on one read, and the remainder of the packet on the next read. I believe this is expected behavior because MSDN says that if no data is available for some period of time, ReadAsync will return with the first byte it receives. However, I think this behavior is somehow related to my problem because when a single byte is missing or out of order, it is "always" that first byte, with the rest of the packet arriving normally.

When the packets are small, the "missing" byte from the front of the packet often (but not always) appears to be delivered in the next read after the remainder of the packet, and this just makes absolutely no sense to me. With larger packets this still happens occasionally, but more often the first byte is just missing when the packets are large.

I've searched far and wide, and have read every SO question I could find on this topic. I found other people with what appears to be a similar problem (ex: SerialPort.BaseStream.ReadAsync missing the first byte ), but nobody with any accepted or even plausible solutions.

Ben Voigt ( http://www.sparxeng.com/blog/software/must-use-net-system-io-ports-serialport ) and others who really seem to know serial comms have recommended the use of ReadAsync on the basestream, and Microsoft's IOT team has also recommended this approach, so I have to believe the approach should work.

Question 1: Why is my code using ReadAsync on a USB Serial BaseStream dropping /scrambling bytes?

Question 2: If ReadAsync cannot be made to reliably return all the bytes received bytes in the correct order, can I just put an async wrapper around the traditional SerialPort.Read and await / loop it so I don't have to poll from a timer? I've read that this is a bad idea, but I've also read that the SerialPort class is internally asynchronous, so perhaps that makes it OK? Or is my only alternative to put this on a worker thread and just let it spend all its time waiting?

My code is below. I have set serialPort1.ReadTimeout = 0; and serialPort1.BaseStream.ReadTimeout = 0; (and I have tried other durations). I have enabled RTS and DTR, and since this is a USB_serial port it should handle handshake internally, and it certainly appears to do so when I read synchronously -- but perhaps that's not true when I read from the BaseStream?

Here is the first method:

// this method works perfectly when called from a timer.
// SerialPort.ReadTimeout must be set to zero for this to work.
// It handles incoming bytes reliably at rates above 1.6 Mbps.

private void ReadSerialBytes()
{
    if (!serialPort1.IsOpen)
        return;

    if (serialPort1.BytesToRead > 0)
    {
        var receiveBuffer = new byte[serialPort1.ReadBufferSize];

        var numBytesRead = serialPort1.Read(receiveBuffer, 0, serialPort1.ReadBufferSize);
        var bytesReceived = new byte[numBytesRead];
        Array.Copy(receiveBuffer, bytesReceived, numBytesRead);

        // Here is where I audit the received data.
        // the NewSerialData event handler displays the 
        // data received (as hex bytes) and writes it to disk.
        RaiseEventNewSerialData(bytesReceived);

        // serialInBuffer is a "thread-safe" global circular byte buffer 
        // The data in serialInBuffer matches the data audited above.
        serialInBuffer.Enqueue(bytesReceived, 0, numBytesRead);
    }
}

Here is the second method, Edited to remove the tail recursion noted by @Lucero. Now I won't run out of memory :) but the original data loss problem, of course, remains.

// This method is called once after the serial port is opened,
// and it repeats until cancelled. 
// 
// This code "works" but periodically drops the first byte of a packet, 
// or returns that byte in the wrong order.
// It occasionally drops several hundred bytes in a row.
private async Task ReadSerialBytesAsync(CancellationToken ct)
{
    while((!ct.IsCancellationRequested) && (serialPort1.IsOpen))
    {
        try
        {
            serialPort1.BaseStream.ReadTimeout = 0;
            var bytesToRead = 1024;
            var receiveBuffer = new byte[bytesToRead];
            var numBytesRead = await serialPort1.BaseStream.ReadAsync(receiveBuffer, 0, bytesToRead, ct);

            var bytesReceived = new byte[numBytesRead];
            Array.Copy(receiveBuffer, bytesReceived, numBytesRead);

             // Here is where I audit the received data.
             // the NewSerialData event handler displays the 
             // data received (as hex bytes) and writes it to disk.
             RaiseEventNewSerialData(bytesReceived);

            // serialInBuffer is a "thread-safe" global circular byte buffer 
            // The data in serialInBuffer matches the data audited above.
            serialInBuffer.Enqueue(receiveBuffer, 0, numBytesRead);
        }
        catch (Exception ex)
        {
            MessageBox.Show("Error in ReadSerialBytesAsync: " + ex.ToString());
            throw;
        }
    }
}

Here is C++ code from the sending system (teensy 3.2 with an ARM chip). It sends a sequence of bytes from 00 through FF, repeated every 50 msec.

 void SendTestData()
 {
    byte asyncTestBuffer[256] = { 0 };
    for (int i = 0; i < 256; i++)
        asyncTestBuffer[i] = i;

    while(true)
    {
    Serial.write(asyncTestBuffer, sizeof(asyncTestBuffer));
    delay(50);
    }
}

The traditional synchronous SerialPort.Read (called from a timer) receives each block completely exactly as expected, with no data loss. It looks like this, over and over:

=====
32 msec => Received 256 bytes 
000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====

Now here is what SerialPort.BaseStream.ReadAsync receives. In another version I appended a terminal packet sequence number to prove that when I see a zero followed by another zero, there's not really an entire missing packet between them. Packet sequence numbers were all present, so the leading byte really does seem to be missing or delivered out of order.

7 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
5 msec => Received 1 bytes 
00
=====
55 msec => Received 1 bytes 
00
=====
4 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
42 msec => Received 1 bytes 
00
=====
5 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
68 msec => Received 1 bytes 
00
=====
7 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
31 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
9 msec => Received 1 bytes 
00
=====
33 msec => Received 1 bytes 
00
=====
10 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
55 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
12 msec => Received 1 bytes 
00
=====
12 msec => Received 1 bytes 
00
=====
15 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
68 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
16 msec => Received 1 bytes 
00
=====
14 msec => Received 256 bytes 
000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====

I've spent a couple of weeks tracking down this problem, which originally manifested itself in bizarre behavior from a product under development. I am pretty sure I must be doing something wrong, but I just can't see it, and at this point I am quite desperate for any thoughts or suggestions!

Answer 1

I finally came up with an answer after stepping through the decompiled source code for the .Net SerialPort class (with resharper installed just Rclick on SerialPort->Navigate->Decompiled Sources ).

Answer #1: The bytes out of order problem was due to an error earlier in my program. I had canceled and restarted the readAsync loop, but I was using the wrong cancellation token so there were two copies of the loop both awaiting readAsync from the serial port. Both were issuing interrupts to return the received data, but of course it was a race condition as to which one got there first.

Answer #2: Note the way I am using the synchronous read method: I don't use the Received event (which doesn't work correctly) or check the number of bytes to read (which is unreliable) or anything like that. I merely set a timeout of zero, attempt to read with a large buffer, and check how many bytes I got back.

When called in this way, the synchronous SerialPort.Read first attempts to fulfill a read request from an internal cache[1024] of data bytes received. If it still doesn't have enough data to meet the request, it then issues a ReadAsync request against the underlying BaseStream using the exact same buffer, (adjusted)offset, and (adjusted)count.

Bottom line: When used the way I am using it, the synchronous SerialPort.Read method behaves exactly like SerialPort.ReadAsync. I conclude that it would probably be fine to put an async wrapper around the synchronous method, and just await it. However, I don't need to do that now that I can read from the basestream reliably.

Update: I now reliably receive more than 3Mbps from my serial port using a Task containing a loop that continuously awaits SerialPort.Basestream.ReadAsync and adds the results to a circular buffer.

Answer 2

I know it's quite some time since question was asked/solved, but noticed it while searching. I have had same kind of "problems" earlier. Nowadays I use a Pipereader over the BaseStream of the serial port to handle reading. This allows me to only clear the incoming buffers when I have a complete message (and receive several messages at the same time). And it seems to perform very well.

Code is something like this:

        var reader = PipeReader.Create(serial.BaseStream);
        while (!token.IsCancellationRequested)
        {
            ReadResult result = await reader.ReadAsync(token);

            // find and handle packets
            // Normally wrapped in a handle-method and a while to allow processing of several packets at once 
            // while(HandleIncoming(result))
            // {
                    result.Buffer.Slice(10); // Moves Buffer.Start to position 10, which we use later to advance the reader
            // }

            // Tell the PipeReader how much of the buffer we have consumed. This will "free" that part of the buffer
            reader.AdvanceTo(result.Buffer.Start, result.Buffer.End);

            // Stop reading if there's no more data coming
            if (result.IsCompleted)
            {
                break;
            }
        }

See the documentation for pipelines here: https://docs.microsoft.com/en-us/dotnet/standard/io/pipelines

Answer 3

I can confirm that the scrambled sequence still persists (or is back?) in NET 6.

I'm writing my first NET 6 desktop application in January 2022 and ran into this problem for the first time. I have been using the SerialPort class for at least 4 or 5 years and never experienced this problem. I use it in almost every app I wirte to communicate with various devices.

I'm just learning that this problem existed for a long time!. The oldest report I a saw was from 2012, ... and it is still around?, seriously?.

Until now the serialport apps that I wrote are based on NET Framework 4.7.2 and older. In this framework SerialPort was part of System.dll. In NET 6 SerialPort is a platform extension moved to System.IO.Ports.dll that has to be installed as a nugget package. Could it be possible that they ported an old, bugged version?

In my test I have the old NET Framework 4.7.2 app sending a string every 20 ms over a phisical port COM3 (no USB adapter). The NET 6 app reading the strings is in the same desktop listening on a second phisical port (COM4). Both ports are linked by a short NULL modem cable, only TX, RX, GND connected. No handshake. This is the scrambled result that, as far as I can tell, is random in nature:

<-- port open with tranmission running already -->
 fox jumps over the lazy Dog - 123456789]
[The quick brown fox jumps over the lazy Dog - 123456789]
[The quick brown[The quick brown fox jumps over the lazy Dog - 123456789]
[The quick brown fox jumps over the lazy Dog - 123456789]
[The quick brown fox jumps over the lazy Dog - 123456789]
<--- many good lines removed for brevity --->
[The quick brown fox jumps over the lazy Dog - 123456789]
[The quick brown fox jumps over he lazy Dog - 1t23456789]
[The quick brown fox jumps over the lazy Dog - 123456789]
<--- many good lines removed for brevity --->
[The quick brown fox jumps over the lazy Dog - 123456789]
[The quick brownfox jumps over  the lazy Dog - 123456789]
[The quick brown fox jumps over the lazy Dog - 123456789]

Notice that the third line has the first three words of the first line!. After that is only one byte out of place. It looks like a slopy double buffer implementation ...

ops! I forgot these bytes!, here you go!

If I open a second good old terminal (Net Framework 4.7.2) and listen to the same stream the output is perfect.

In the NET 6 project I'm using exactly the same class I wrote long time ago to encapsulate the SerialPort functionality to use it from project to the next.

Until now I subscribed to the SerialPort.DataReceived event and then in the event handler the reading goes like this inside a Task (started but not waited by the event handler):

var bytesToRead = _serialPort.BytesToRead;
byte[] Data = new Byte[bytesToRead];
int received = _serialPort.Read(Data, 0, bytesToRead);
... notify the class user.

I'll test the work arround suggested here ...