LIVE555 how to use h264 framer class to get nal units for ffmpeg

Question

I'm trying to create a small app which will save frames from inoming h264 stream. I took a testRTSP programm as example and made several changes in DummySink::afterGettingFrame function to decode frames with the help of ffmpeg library. As I understand from frameSize, my first two frames are SPS units, so I concatenate them with my third frame and then send new big frame to ffmpeg decoder. But that doesnt work. ffmpeg tells me that my first frame is too big for SPS and then it tells me that there is no frame... I dont know what I need to change here.

void DummySink::afterGettingFrame(unsigned frameSize, unsigned numTruncatedBytes,
struct timeval presentationTime, unsigned /*durationInMicroseconds*/)
{
u_int8_t start_code[4] = { 0x00, 0x00, 0x00, 0x01 };
int stCodeLen = 4;

if (frameSize == 50)
{
    //add start code
    memcpy(bufferWithStartCode, start_code, stCodeLen);
    shiftPtr += stCodeLen;
    memcpy(bufferWithStartCode + shiftPtr, fReceiveBuffer, frameSize);
    shiftPtr += frameSize;
}
else if (frameSize == 4)
{
    memcpy(bufferWithStartCode + shiftPtr, fReceiveBuffer, frameSize);
    shiftPtr += frameSize;
}
else
{
    if (shiftPtr == 0)
    {
        memcpy(bufferWithStartCode, start_code, stCodeLen);
        shiftPtr += stCodeLen;
    }
    memcpy(bufferWithStartCode + shiftPtr, fReceiveBuffer, frameSize);
    avpkt.size = frameSize + shiftPtr;
    avpkt.data = bufferWithStartCode;
    shiftPtr = 0;
    if (!avcodec_send_packet(cContext, &avpkt))
    {
        envir() << "error sending to decoder";

    }
    if (!avcodec_receive_frame(cContext, picture))
    {
        envir() << "error rx from decoder";
    }
    if (picture)
    {
        FILE *f;
        char buffer[32]; // The filename buffer.
        snprintf(buffer, sizeof(char) * 32, "file%i.txt", frame_num);
        f = fopen(buffer, "w");
        fprintf(f, "P5\n%d %d\n%d\n", fSubsession.videoWidth(), fSubsession.videoHeight(), 255);
        for (int i = 0;i < fSubsession.videoHeight();i++)
            fwrite(picture->data[0] + i * (picture->linesize[0]), 1, fSubsession.videoWidth(), f);
        fclose(f);
    }
}

envir() << frameSize << "\n";


frame_num++;

// Then continue, to request the next frame of data:
continuePlaying();

Answer 1

I've found solution for my problem.

There is function void DummySink::afterGettingFrame(...) In "testRTSP" example of live555. All I was need to do is to assembly my frame everytime when function got frame:

[start_code sps pps start_code frame_data]

frame_data is fReceiveBuffer at this point. start_code is char array [0,0,0,1].

And push new data to ffmpeg decoder:

m_packet.size = frameBufSize + frameSize; // size of assembled frame
m_packet.data = frameBuf; // assembled frame

if (avcodec_send_packet(m_decoderContext, &m_packet) != 0)
{
    envir() << "error in sending packet to decoder" << "\n";
}
if (avcodec_receive_frame(m_decoderContext, pFrame) == 0)

No additional settings for decoderContext! Just init everything like in tutorials ( http://dranger.com/ffmpeg/ - this is up-to-date tutorials for ffmpeg c++ library) and you good to go. I used memcpy to unite data in one big array. memcpy(frameBuf, startCode, 4); frameBufSize += 4;

for (int i = 0; i < numSPropRecords; i++)
{
    memcpy(frameBuf + frameBufSize, sPropRecords[i].sPropBytes, sPropRecords[i].sPropLength);
    frameBufSize += sPropRecords[i].sPropLength;
}

memcpy(frameBuf + frameBufSize, startCode, 4);
frameBufSize += 4;
memcpy(frameBuf + frameBufSize, fReceiveBuffer, frameSize);

m_packet.size = frameBufSize + frameSize;
m_packet.data = frameBuf;

You can get sps and pps data from subsession (check "openRTSP" for detailed example or "H264orH264FileSink.h")

spsppsunits = subsession.fmtp_spropparametersets();
sPropRecords = parseSPropParameterSets(spsppsunits, numSPropRecords);

UPDATE 1:

After some time I found errors in my approach to ffmpeg. Sometimes ffmpeg decoder wont work with H264 stream if you wont supply extradata information with sps and pps units. So,

m_decoderContext->extradata =  (uint8_t*)av_malloc(100 + AV_INPUT_BUFFER_PADDING_SIZE);
int extraDataSize = 0;
for (int i = 0; i < numSPropRecords; i++)
{
    memcpy(m_decoderContext->extradata + extraDataSize, startCode, 4);
    extraDataSize += 4;
    memcpy(m_decoderContext->extradata + extraDataSize, sPropRecords[i].sPropBytes, sPropRecords[i].sPropLength);
    extraDataSize += sPropRecords[i].sPropLength;
}
m_decoderContext->extradata_size = extraDataSize;

After that you dont need to supply frame with sps and pps data every time, only start code is needed.

Answer 2

Your code does not show how you initialize the codec but the SPS and PPS should not go in the packet. Instead they should be passed to the codec via the extradata field of AVCodecContext upon initialization. And then you would only pass the actual frame NALs to the decoder to obtain decoded pictures.

Id suggest that you either initialize the decoder upon the reception of the first SPS or event from the SDP data in response to the DESCRIBE .