ffmpeg based multi threaded c++ application fails on decoding

Question

I am using remuxing example from ffmpeg sources as reference. I wrote a multi-threaded application based on boost threads to perform a codec copy and remux using ffmpeg API. That works fine . The problem arises when I try to decode the frame

"

 ret = avcodec_decode_video2(dec_ctx, frame, &got_frame, &pkt);
            if (ret < 0) {
                av_log(NULL, AV_LOG_ERROR, "Error decoding video %s\n",av_make_error_string(errorBuff,80,ret));
                return -1;
            }"

I need the decoded frame to convert it to Opencv Mat object. For a single instance this code works fine. But as soon as I run multiple threads I start getting decoding errors like these

   left block unavailable for requested intra mode at 0 0
[h264 @ 0x7f9a48115100] error while decoding MB 0 0, bytestream 1479
[h264 @ 0x7f9a480825e0] number of reference frames (0+2) exceeds max (1; probably corrupt input), discarding one
[h264 @ 0x7f9a480ae680] error while decoding MB 13 5, bytestream -20
[h264 @ 0x7f9a48007700] number of reference frames (0+2) exceeds max (1; probably corrupt input), discarding one
[h264 @ 0x7f9a48110340] top block unavailable for requested intra4x4 mode -1 at 31 0
[h264 @ 0x7f9a48110340] error while decoding MB 31 0, bytestream 1226
[h264 @ 0x7f9a48115100] number of reference frames (0+2) exceeds max (1; probably corrupt input), discarding one
[h264 @ 0x7f9a480825e0] top block unavailable for requested intra4x4 mode -1 at 4 0
[h264 @ 0x7f9a480825e0] error while decoding MB 4 0, bytestream 1292
[h264 @ 0x7f9a480ae680] number of reference frames (0+2) exceeds max (1; probably corrupt input), discarding one

All variables used by ffmpeg api are declared local to the thread function. I am not sure how ffmpeg frame allocs or context allocs work.

any help in making the decoding process multi-threaded ?

Update: I have included ff_lockmgr

static int ff_lockmgr(void **mutex, enum AVLockOp op)
{
   pthread_mutex_t** pmutex = (pthread_mutex_t**) mutex;
   switch (op) {
   case AV_LOCK_CREATE:
      *pmutex = (pthread_mutex_t*) malloc(sizeof(pthread_mutex_t));
       pthread_mutex_init(*pmutex, NULL);
       break;
   case AV_LOCK_OBTAIN:
       pthread_mutex_lock(*pmutex);
       break;
   case AV_LOCK_RELEASE:
       pthread_mutex_unlock(*pmutex);
       break;
   case AV_LOCK_DESTROY:
       pthread_mutex_destroy(*pmutex);
       free(*pmutex);
       break;
   }
   return 0;
}

and initialized it as well " av_lockmgr_register(ff_lockmgr); " Now the video is being decoded in all threads BUT the images saved from the decoded frame using FFMPEG AVFrame to OpenCv Mat conversion and imwrite results in garbled (mixed) frame. Part of the frame is from one camera and rest is from another or the image doesnt make any sense at all.

Answer 1

Not every format decoder supports multiple threads, and even for the decoders which support it, it might not be supported for a particular file.

For example, consider a MPEG4 file with a single keyframe at the beginning, followed by P frames. In this case every next frame depends on previous, and using multiple threads would not likely produce any benefits.

In my app I had to disable multithreaded encoders because of that.