使用Ffmpeg传输SEI

Question

I need to transport some data from drone through the help of Ffmpeg.The data includes frame data and some other parameters such as the timing roll/pitch/height/direction of the drone,and when I get the image of the frame,the corresponding parameters should be also pulled out,So I dicede to send these parameters as SEI. Then the questions coming,after sending sps pps,I send my own SEI packet，eg:

 if(pkt->size>10&&pkt->data[0] ==0
       &&pkt->data[1] ==0
       &&pkt->data[2] ==0
       &&pkt->data[3] ==1
       &&(pkt->data[4] == 103 || pkt->data[4] == 104)){
        i++;
    }
    AVPacket *newPacket = nullptr;
    // 0 0 0 1 6 8*16
    if(i == 2){
        i = 0;
        newPacket = (AVPacket *)av_malloc(sizeof(AVPacket));
        av_init_packet(newPacket);
        newPacket->data = new uint8_t[8];
        newPacket->data[0] = 0;
        newPacket->data[1] = 0;
        newPacket->data[2] = 0;
        newPacket->data[3] = 1;
        newPacket->data[4] = 6;
        newPacket->data[5] = 123;
        newPacket->data[6] = 134;
        newPacket->data[7] = 128;
        newPacket->size = 8;
    }
 av_write_frame(*it , pkt);
        av_free_packet(pkt);
        if(newPacket){
            av_write_frame(*it , newPacket);
            av_free_packet(newPacket);
        }

But,At the receiving terminate,I only find the API:av_read_frame.The API just decode every complete frame from AVFormatContext.My SEI go nothing! Besides,I also trying to put my parameters in side_data or metadate of AVFrame,but after rtp tansporting,received AVFrame's side_data and metadata is 0x00 again. Could someone give me some train of thought?

Answer 1

At the receiving terminal, did you check if the AVPackets you obtained using av_read_frame() has your SEI message on their data? My approach to save SEI for each frame was similiar to yours except the encoding part. My steps are:

1. First, I encoded frames using x264 library (I don't think this would make a difference)
2. Then, like you I fwrite() a packet then a SEI (you don't have to explicity declare a AVPacket for SEI and use packet->data for SEI, simply declare uchar* or uint8_t*)
3. After using av_read_frame(), the SEI message appears on packet->data along with encoded frame.
4. I obtain SEI message from packet->data just after the encoded frame's last byte. (Actually, you cannot determine the last byte of encoded frame. But you can do a trick and indicate its length on SEI)
5. Lastly, without modifying the packet->data, I decode the frame with the usual way.