Live video streaming over UDP

Question

I am creating an application which records the desktop screen and sends it over a network. I was using TCP and it was working, however there was huge frame stutters even when doing it on the same machine. When the screen would change it would require more data to be sent, this usually caused the TCP client to take an abornal amount of time sending the data

Client

byte[] encoded = Encoder.Encode(frame); // Takes in a bitmap image and only keeps
// the part of the image that has changed

byte[] compressed = DataCompressor.Compress(encoded); // GZIP Compresses image

byte[][] slices = ByteManipulator.SliceBytes(compressed, 15000); // Divides the
// image slices that are 15000 bytes in length

foreach (byte[] slice in slices)
{
     SendTo(slice, HostIPEP); // Sends to the server (The main issue)
}

The issue with the above code is that data does not receive in the correct order, due to it being UDP. How does one get around this issue to stream video like this over UDP?

Answer 1

Well, the stutters come from the huge amount of data to transfer, at the begging, and when something change in screen content.

I think if you use professional encoder for image(screen content), it should be much better, for example, x264/x265/av1 encoder, especially AV1 Real-Time Screen Content Coding .

For the transport, recommend to use WebRTC server, such as SRS or MediaSoup etc, please read more detail from this post .

If use C# to build the app, there is also some native binding to use WebRTC in C#.