Creating a usable H.264 video file

Question

I am trying to use libavcodec to generate an mp4 video file from individual frames. Each input frame is a qt QImage , and the output file is written to using the Qt QFile class.

I've done this through a VideoTarget class which opens the given 'target' file when initialized, records frames when addFrame(image) is called, and then saves/closes the file when its destructor is called.

The class has the following fields:

AVCodec* m_codec = nullptr;
AVCodecContext *m_context = nullptr;
AVPacket* m_packet = nullptr;
AVFrame* m_frame = nullptr;

QFile m_target;

And looks like this:

VideoTarget::VideoTarget(QString target, QObject *parent) : QObject(parent), m_target(target)
{
    // Find video codec
    m_codec = avcodec_find_encoder_by_name("libx264rgb");
    if (!m_codec) throw std::runtime_error("Unable to find codec.");

    // Make codec context
    m_context = avcodec_alloc_context3(m_codec);
    if (!m_context) throw std::runtime_error("Unable to allocate codec context.");

    // Make codec packet
    m_packet = av_packet_alloc();
    if (!m_packet) throw std::runtime_error("Unable to allocate packet.");

    // Configure context
    m_context->bit_rate = 400000;
    m_context->width = 1280;
    m_context->height = 720;
    m_context->time_base = (AVRational){1, 60};
    m_context->framerate = (AVRational){60, 1};
    m_context->gop_size = 10;
    m_context->max_b_frames = 1;
    m_context->pix_fmt = AV_PIX_FMT_RGB24;

    if (m_codec->id == AV_CODEC_ID_H264)
        av_opt_set(m_context->priv_data, "preset", "slow", 0);

    // Open Codec
    int ret = avcodec_open2(m_context, m_codec, nullptr);
    if (ret < 0) {
        throw std::runtime_error("Unable to open codec.");
    }

    // Open file
    if (!m_target.open(QIODevice::WriteOnly))
        throw std::runtime_error("Unable to open target file.");

    // Allocate frame
    m_frame = av_frame_alloc();
    if (!m_frame) throw std::runtime_error("Unable to allocate frame.");

    m_frame->format = m_context->pix_fmt;
    m_frame->width = m_context->width;
    m_frame->height = m_context->height;
    m_frame->pts = 0;

    ret = av_frame_get_buffer(m_frame, 24);
    if (ret < 0) throw std::runtime_error("Unable to allocate frame buffer.");
}

void VideoTarget::addFrame(QImage &image)
{
    // Ensure frame data is writable
    int ret = av_frame_make_writable(m_frame);
    if (ret < 0) throw std::runtime_error("Unable to make frame writable.");

    // Prepare image
    for (int y = 0; y < m_context->height; y++) {
        for (int x = 0; x < m_context->width; x++) {
            auto pixel = image.pixelColor(x, y);
            int pos = (y * 1024 + x) * 3;
            m_frame->data[0][pos] = pixel.red();
            m_frame->data[0][pos + 1] = pixel.green();
            m_frame->data[0][pos + 2] = pixel.blue();
        }
    }

    m_frame->pts++;

    // Send the frame
    ret = avcodec_send_frame(m_context, m_frame);
    if (ret < 0) throw std::runtime_error("Unable to send AV frame.");

    while (ret >= 0) {
        ret = avcodec_receive_packet(m_context, m_packet);
        if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
            return;
        else if (ret < 0) throw std::runtime_error("Error during encoding.");

        m_target.write((const char*)m_packet->data, m_packet->size);
        av_packet_unref(m_packet);
    }
}

VideoTarget::~VideoTarget()
{
    int ret = avcodec_send_frame(m_context, nullptr);
    if (ret < 0) throw std::runtime_error("Unable to send AV null frame.");

    while (ret >= 0) {
        ret = avcodec_receive_packet(m_context, m_packet);
        if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
            return;
        else if (ret < 0) throw std::runtime_error("Error during encoding.");

        m_target.write((const char*)m_packet->data, m_packet->size);
        av_packet_unref(m_packet);
    }

    // Magic number at the end of the file
    uint8_t endcode[] = { 0, 0, 1, 0xb7 };
    m_target.write((const char*)endcode, sizeof(endcode));
    m_target.close();

    // Free codec stuff
    avcodec_free_context(&m_context);
    av_frame_free(&m_frame);
    av_packet_free(&m_packet);
}

When used, the class seems to work, and data is written to the file, except I am unable to play back the resulting file in any application.

My main suspect is these lines:

    // Prepare image
    for (int y = 0; y < m_context->height; y++) {
        for (int x = 0; x < m_context->width; x++) {
            auto pixel = image.pixelColor(x, y);
            int pos = (y * 1024 + x) * 3;
            m_frame->data[0][pos] = pixel.red();
            m_frame->data[0][pos + 1] = pixel.green();
            m_frame->data[0][pos + 2] = pixel.blue();
        }
    }

The libavcodec documentation was extremely vague regarding the layout of image data, so I effectively had to guess and be happy with the first thing that didn't crash, so chances are I'm writing this incorrectly. There's also the issue of size mismatch between my pixel color data calls (giving int values) and the 24-bits-per-pixel RGB format I have selected.

How do I tweak this code to output actual, functioning video files?

Answer 1

The libavcodec documentation was extremely vague regarding the layout of image data

This is because every codec is different. I recommend that you use yuv420p and not RGB24. Many players can not play h264 rgb. You can use libswscale to convert between.

Next, What format is the stream you are producing? Annex B can be played directly, but if you are using extradata + NALU size (AVCC), you will need to wrap the stream in a container.

Lastly, why use libavcodec? libx264 provides a cleaner API in my opinion. Unless you play to switch codecs later, avoid the abstraction.