FFmpeg: Decoding AVPackets received from UDP socket

Question

I am trying to stream video frames encoded with FFmpeg from an Unity game to a client UI via UDP. To be specific, I am sending AVPackets (which contain the compressed frames, as far as I understand) from the server, which are then, on the client side, extracted as follows:

inline UDPpacket* SDLGameClient::receiveData()
{
    if(SDLNet_UDP_Recv(socket, packet))
        return packet;
    else
        return NULL;
}

int main()
{
    ...
    // Initialize UDP
    ...
    UDPpacket *udpPacket;

    int i = 0;

    for(;;)
    {
        udpPacket = client->receiveData();

        if(udpPacket != nullptr)
        {
            i++;
            std::cout << "Packet " << i << " received!" << std::endl;

            AVPacket packet;
            av_init_packet(&packet);

            packet.data = new uint8_t[udpPacket->len];
            memcpy(packet.data, udpPacket->data, udpPacket->len);
            packet.size = udpPacket->len;

            ...

To realize networking, I use the SDL_net library. Fragmenting, sending and receiving the packets seem to work without any problem. My question is, how do I decode the incoming AVPackets and stream the frames recorded in Unity to the client? I am aware that I need to use avcodec_send_packet and avcodec_receive_frame for decoding (since avcodec_decode_video2 , that is used in many example decoding code, is deprecated), but when I do something like this:

int ret = avcodec_send_packet(codecContext, &packet);
if(ret < 0 || ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
    std::cout << "avcodec_send_packet: " << ret << std::endl;
else
{
    while(ret >= 0)
    {
        ret = avcodec_receive_frame(codecContext, frame);
        if(ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
            std::cout << "avcodec_receive_frame: " << ret << std::endl;

        std::cout << "Frame: " << codecContext->frame_number << std::endl;
    }
}

av_packet_unref(packet);

ret always returns a negative value (-22), so perhaps something is wrong with the AVPackets, or I'm sending the frames way too fast, I really have no clue:/

Thanks in advance for any help!

Answer 1

-22 means -EINVAL , Invalid argument . You can normally check the ffmpeg error log messages for more details.

My guess is that your packets are still fragmented, but avcodec_send_packet() expects AVPacket to always contain a full frame. Depending on the codec involved you should consider using an AVParser to recover the framing or frame the packets before UDP fragmentation, so you can recover the framing by yourself.

Answer 2

Firstly sorry, i cannot add comment on this because of reputation. I am tyring to do exactly the same. Also i am getting exact same error. However, i didnt get how to add av_parser_parse2() function. It also requires AVCodecparseContext which is also need to be send separately from sender. Would you be able to share the part you have added av_parser_parse2() function?