Converting an AVFrame to QImage with conversion of pixel format

Question

I need to extract frames of videos to images in my QT application. I don't know in advance the pixel format of the source videos/frames (yuv, rgb...), But I need to obtain a reliable image pixel format such that I can consistently handle the images later. I'm using the ffmpeg libraries to get the frames, which are already decoded. I'm trying to avoid using deprecated functions, and I need to optimize for speed.

I tried implementing this example: https://stackoverflow.com/a/42615610/7360943 which

1. Converts the original frame to a rgb frame

2. Creates a QImage from this second frame's data.

However, this has two main problems: it uses the deprecated avpicture_alloc function, and also doesn't free the allocated memory , which causes my app to quickly break down, using more, and more RAM until it crashes when multiple thousand images need to be taken. I haven't found a way to solve either problem independently, for I don't know what to use instead of avpicture_alloc, and if I use avpicture_free it actually frees the data underlying the QImage, which breaks the QImage.

I tried the following instead, directly passing the QImage preallocated data to sws_scale, which works great most of the time :

// we will convert the original color format to rgb24
SwsContext* img_convert_ctx = sws_getContext(
                                 pFrame->width,
                                 pFrame->height,
                                 (AVPixelFormat)pFrame->format,
                                 pFrame->width,
                                 pFrame->height,
                                 AV_PIX_FMT_RGB24,
                                 SWS_BICUBIC, NULL, NULL, NULL);

QImage image(pFrame->width,
             pFrame->height,
             QImage::Format_RGB888);

int rgb_linesizes[8] = {0};
rgb_linesizes[0] = 3*pFrame->width;

sws_scale(img_convert_ctx,
            pFrame->data,
            pFrame->linesize, 0,
            pFrame->height,
            (uint8_t *[]){image.bits()},
            rgb_linesizes);

ffmpeg::sws_freeContext(img_convert_ctx);

The problem is that for some specific videos , it outputs some weird images that look somewhat black and white (and which maybe seems to show an offset of 1 between input width and output width...? I haven't been able to fully interpret what could cause this): See the reference image, how it should look:

and the problematic grey-ish image:

So what's the problem in my code that causes it to behave well most of the time, but not work as intended in some specific cases? Or how could this be done otherwise?

Answer 1

I figured it out in the end, using a manually allocated buffer, which isn't very clean C++ code but works faster and without deprecated calls. It's not possible to pass image.bits directly to sws_scale because QImages are minimum 32 bit aligned ( https://doc.qt.io/qt-5/qimage.html#scanLine ), meaning that depending on the image width, there is "empty space" in memory at the end of each line, which sws_scale does not skip/take into account. It's too bad because we now have two memory copying operations, in sws_scale and memcpy, instead of one, but I have not found a better way.

I still have a problem with the buffer allocation size, needing at least 64 extra bytes for no reason that I can make sense of, but otherwise we sometimes got segmentation faults. It might be due to how memcpy works, copying entire 32 or 64 byte blocks... But anyways here's the new implementation :

(NB : I import ffmpeg functions under a dedicated namespace, explaining the ffmpeg:: before each call)

QImage getQImageFromFrame(const ffmpeg::AVFrame* pFrame) const
{
    // first convert the input AVFrame to the desired format

    ffmpeg::SwsContext* img_convert_ctx = ffmpeg::sws_getContext(
                                     pFrame->width,
                                     pFrame->height,
                                     (ffmpeg::AVPixelFormat)pFrame->format,
                                     pFrame->width,
                                     pFrame->height,
                                     ffmpeg::AV_PIX_FMT_RGB24,
                                     SWS_BICUBIC, NULL, NULL, NULL);
    if(!img_convert_ctx){
        qDebug() << "Failed to create sws context";
        return QImage();
    }

    // prepare line sizes structure as sws_scale expects
    int rgb_linesizes[8] = {0};
    rgb_linesizes[0] = 3*pFrame->width;

    // prepare char buffer in array, as sws_scale expects
    unsigned char* rgbData[8];
    int imgBytesSyze = 3*pFrame->height*pFrame->width;
    // as explained above, we need to alloc extra 64 bytes
    rgbData[0] = (unsigned char *)malloc(imgBytesSyze+64); 
    if(!rgbData[0]){
        qDebug() << "Error allocating buffer for frame conversion";
        free(rgbData[0]);
        ffmpeg::sws_freeContext(img_convert_ctx);
        return QImage();
    }
    if(ffmpeg::sws_scale(img_convert_ctx,
                pFrame->data,
                pFrame->linesize, 0,
                pFrame->height,
                rgbData,
                rgb_linesizes)
            != pFrame->height){
        qDebug() << "Error changing frame color range";
        free(rgbData[0]);
        ffmpeg::sws_freeContext(img_convert_ctx);
        return QImage();
    }

    // then create QImage and copy converted frame data into it

    QImage image(pFrame->width,
                 pFrame->height,
                 QImage::Format_RGB888);

    for(int y=0; y<pFrame->height; y++){
        memcpy(image.scanLine(y), rgbData[0]+y*3*pFrame->width, 3*pFrame->width);
    }

    free(rgbData[0]);
    ffmpeg::sws_freeContext(img_convert_ctx);
    return image;
}