Media Foundation Webcam live capture freezes in low light condition

Question

We are building a video communication software. We are using Media Foundation to obtain the live Stream. We use the IMFSourceReadder to perform the capture.

The sequence of call looks like:

hr = pAttributes->SetString(MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE_VIDCAP_SYMBOLIC_LINK, m_pwszSymbolicLink);

hr = MFCreateDeviceSourceActivate(pAttributes, &avdevice);

hr = avdevice->ActivateObject(__uuidof(IMFMediaSource), (void**) &m_mediaSource);

hr = m_mediaSource->CreatePresentationDescriptor(&pPD);

hr = pPD->GetStreamDescriptorByIndex(m_streamIdx, &fSelected, &pSD);

hr = 

// we select the best native MediaType enumerating the source reader
pHandler->SetCurrentMediaType(m_bestNativeType);

hr = pAttributes->SetUINT32(MF_READWRITE_DISABLE_CONVERTERS, FALSE);
hr = pAttributes->SetUINT32(MF_SOURCE_READER_ENABLE_ADVANCED_VIDEO_PROCESSING, TRUE);

hr = MFCreateSourceReaderFromMediaSource(m_mediaSource, pAttributes, &m_reader);    

Then we start to read the frame SYNCHRONOUSLY in a separate thread using

m_reader->ReadSample()

When we need to stop the device or reconfigure it, we stop the thread (by setting an flag and exiting the thread). We call the following

hr = m_mediaSource->Stop();
m_mediaSource->Shutdown();
SafeRelease(&m_mediaSource);
SafeRelease(&m_reader);

The software can be out ouf call. There, it captures the webcam video in VGA format and display it on screen. In call, it selects the best capture format depending on the negociated call quality and restarts the capture.

The issues that we are experiencing are the following: some cameras freeze sometimes in low light conditions (low fps output). It can happen right away at the beginning of the call or during the call.

When it freezes, one of the two things can happen (not sure which one)

• m_reader->ReadSample() fails repetitively with MF_E_OPERATION_CANCELLED error code

• m_reader->ReadSample() returns often producing more than 80 frames per seconds producing same frozen image.

When we hang up the device is reconfigured back to VGA capture and works fine.

Does someone struggled with Media Foundation on the same issue?

Answer 1

You wrote that web camera "freez" - produce low frame rate while capture image with low light condition. The result of it that controller of web camera take more time on exposition of photo matrix in automatic mode. It allows improve quality of image by increasing frame duration. So, it is special feature of hardware part. it is possible to switch such behavior of camera from auto mode on manual mode of parameter

Code::Result VideoCaptureDevice::setParametrs(CamParametrs parametrs){
ResultCode::Result result = ResultCode::VIDEOCAPTUREDEVICE_SETPARAMETRS_ERROR;

if(pLocalSource)
{
    unsigned int shift = sizeof(Parametr);

    Parametr *pParametr = (Parametr *)(&settings);

    Parametr *pPrevParametr = (Parametr *)(&prevParametrs);

    CComPtrCustom<IAMVideoProcAmp> pProcAmp;


    HRESULT hr = pLocalSource->QueryInterface(IID_PPV_ARGS(&pProcAmp));

    if (SUCCEEDED(hr))
    {
        for(unsigned int i = 0; i < 10; i++)
        {
            if(pPrevParametr[i].CurrentValue != pParametr[i].CurrentValue || pPrevParametr[i].Flag != pParametr[i].Flag)
                hr = pProcAmp->Set(VideoProcAmp_Brightness + i, pParametr[i].CurrentValue, pParametr[i].Flag);

        }
    }
    else
    {
        result = ResultCode::VIDEOCAPTUREDEVICE_SETPARAMETRS_SETVIDEOPROCESSOR_ERROR;

        goto finish;
    }

    CComPtrCustom<IAMCameraControl> pProcControl;

    hr = pLocalSource->QueryInterface(IID_PPV_ARGS(&pProcControl));

    if (SUCCEEDED(hr))
    {
        for(unsigned int i = 0; i < 7; i++)
        {
            if(pPrevParametr[10 + i].CurrentValue != pParametr[10 + i].CurrentValue || pPrevParametr[10 + i].Flag != pParametr[10 + i].Flag)
            hr = pProcControl->Set(CameraControl_Pan+i, pParametr[10 + i].CurrentValue, pParametr[10 + i].Flag);                    
        }

    }
    else
    {
        result = ResultCode::VIDEOCAPTUREDEVICE_SETPARAMETRS_SETVIDEOCONTROL_ERROR;

        goto finish;
    }

    result = ResultCode::OK;

    prevParametrs = parametrs.settings;
}finish:
if(result != ResultCode::OK)
    DebugPrintOut::getInstance().printOut(L"VIDEO CAPTURE DEVICE: Parametrs of video device cannot be set!!!\n");

return result;
}

where:

struct Parametr
{
long CurrentValue;

long Min;

long Max;

long Step;

long Default; 

long Flag;

Parametr();
};

  struct CamParametrs
  {
    Parametr Brightness;
    Parametr Contrast;
    Parametr Hue;
    Parametr Saturation;
    Parametr Sharpness;
    Parametr Gamma;
    Parametr ColorEnable;
    Parametr WhiteBalance;
    Parametr BacklightCompensation;
    Parametr Gain;


    Parametr Pan;
    Parametr Tilt;
    Parametr Roll;
    Parametr Zoom;
    Parametr Exposure;
    Parametr Iris;
    Parametr Focus;
  };

More code you can find on site:
Capturing Live-video from Web-camera on Windows 7 and Windows 8

However, using of IMFSourceReader can be not effective. Media Foundation model uses async interaction - after sending the request into the media source code must listen responding from media source with new frame or some other info. Method with direct calling m_reader->ReadSample() cannot be effective - you faced with it. Method m_reader->ReadSample() can be effective with reading frames from video file while delay can be very low, but for web camera I can advice use topology - session binding, like in my code Capturing Live-video from Web-camera on Windows 7 and Windows 8

Regards, Evgeny Pereguda

Answer 2

The question description leaves an impression that you do things in a somewhat chaotic way and the resulting freeze is not necessarily caused by Media Foundation or camera.

Use of media source and source reader are certainly the right way to access a camera and it provides efficient way to capture video, both synchronously and asynchronously.

However, your incomplete code snippets show that you create a media source, then source reader, and then you keep dealing with media source directly. Well, you are not supposed to do this. Once you created a source reader, it will manage media source for you: you don't need Stop , Shutdown calls. Your calling that and other methods might bring confusion that results in incorrect source reader behavior.

That is, either you deal with a media source, or you plug it into Media Session or Source Reader and use this higher level API.

Also note that if/when you experience a freeze, you are interested to break in with debugged and locate threads that indicate freeze position.