尝试在媒体基础编码中使用 MFT

Question

The idea is to use a Media Foundation Transform, such as the Video Stabilization MFT while transcoding a video with Media Foundation.

When not using an MFT, the code works fine.

• Create IMFSourceReader for the source file - OK
• Create IMFSinkWriter for the target file - OK
• Add a stream to the writer describing the Video - OK
• Add the audio stream - OK
• Set input types for video and audio - OK
• Loop to read samples and send them to the sink writer, OK.

When using the MFT, these are the facts. To create the MFT (error checking removed):

CComPtr<IMFTransform> trs;
trs.CoCreateInstance(CLSID_CMSVideoDSPMFT);
std::vector<DWORD> iids;
std::vector<DWORD> oods;
DWORD is = 0, os = 0;
hr = trs->GetStreamCount(&is, &os);
iids.resize(is);
oods.resize(os);
hr = trs->GetStreamIDs(is, iids.data(), os, oods.data());
CComPtr<IMFMediaType> ptype;
CComPtr<IMFMediaType> ptype2;
MFCreateMediaType(&ptype);
MFCreateMediaType(&ptype2);
SourceVideoType->CopyAllItems(ptype);
SourceVideoType->CopyAllItems(ptype2);

ptype->SetUINT32(MF_VIDEODSP_MODE, MFVideoDSPMode_Stabilization);
//                LogMediaType(ptype);
ptype2->SetUINT32(MF_VIDEODSP_MODE, MFVideoDSPMode_Stabilization);
//              LogMediaType(ptype2);
hr = trs->SetInputType(iids[0], ptype, 0);
auto hr2 = trs->SetOutputType(oods[0], ptype2, 0);
if (SUCCEEDED(hr) && SUCCEEDED(hr2))
{
    VideoStabilizationMFT = trs;
}

This code works - the MFT is successfully configured. However, in my sample processing loop:

// pSample = sample got from the reader
CComPtr<IMFSample> pSample2;
LONGLONG dur = 0, tim = 0;
pSample->GetSampleDuration(&dur);
pSample->GetSampleTime(&tim);

trs->ProcessInput(0, pSample, 0);
MFT_OUTPUT_STREAM_INFO si = {};
trs->GetOutputStreamInfo(0, &si);

// Create pSample2 
MFCreateSample(&pSample2);
CComPtr<IMFMediaBuffer> bb;
MFCreateMemoryBuffer(si.cbSize, &bb);
pSample2->AddBuffer(bb);

DWORD st = 0;
hr = trs->ProcessOutput(0, 1, &db, &st);

This last call fails initially with MF_E_TRANSFORM_NEED_MORE_INPUT , I can understand that the MFT needs more than one sample to achieve stabilizization, so I skip this sample for the writer.

When the call succeeds, I get a sample with no time or duration. Even If I set the time and duration manually, the sink writer fails with E_INVALIDARG.

What do I miss?

Answer 1

With this source code I provide, the sink writer returns S_OK :

VideoStabilizationMFT

if Microsoft is reading this, what are these guids from CLSID_CMSVideoDSPMFT ?

Guid : 44A4AB4B-1D0C-4181-9293-E2F37680672E : VT_UI4 = 4

Guid : 8252D735-8CB3-4A2E-A296-894E7B738059 : VT_R8 = 0.869565

Guid : 9B2DEAFE-37EC-468C-90FF-024E22BD6BC6 : VT_UI4 = 0

Guid : B0052692-FC62-4F21-A1DD-B9DFE1CEB9BF : VT_R8 = 0.050000

Guid : C8DA7888-14AA-43AE-BDF2-BF9CC48E12BE : VT_UI4 = 4

Guid : EF77D08F-7C9C-40F3-9127-96F760903367 : VT_UI4 = 0

Guid : F67575DF-EA5C-46DB-80C4-CEB7EF3A1701 : VT_UI4 = 1

Microsoft, are you serious ?

according to this documentation : https://docs.microsoft.com/en-us/windows/win32/medfound/video-stabilization-mft

On Win10 :

• Video Stabilization MFT is MF_SA_D3D11_AWARE, the documentation does not speak about this
• Video Stabilization MFT can fallback to software mode, the documentation does not speak about this (see MF_SA_D3D11_AWARE)
• Video Stabilization MFT has dynamic format change, the MFT_SUPPORT_DYNAMIC_FORMAT_CHANGE is not present on IMFTransform::GetAttributes
• Video Stabilization MFT implements IMFGetService/IMFRealTimeClientEx/IMFShutdown, it is not in the documentation
• Video Stabilization MFT only handles MFVideoFormat_NV12, the documentation speak about MEDIASUBTYPE_YUY2
• The documentation tells to include Camerauicontrol.h, seriously...

Having said that, this MTF is really good doing stabilization...

Answer 2

This is strange, you set attribute on IMFMediaType, not on Video Stabilization MFT :

ptype->SetUINT32(MF_VIDEODSP_MODE, MFVideoDSPMode_Stabilization);
ptype2->SetUINT32(MF_VIDEODSP_MODE, MFVideoDSPMode_Stabilization);

Should be :

Call IMFTransform::GetAttributes on the video stabilization MFT to get an IMFAttributes pointer.
Call IMFAttributes::SetUINT32 to set the attribute.

MF_VIDEODSP_MODE attribute