使用MediaCapture直播

Question

For a project I am trying to create a video calling app. I'm doing this for the Universal Windows Platform, so I figured I could use the MediaCapture class. This class does have the StartRecordToCustomSinkAsync() method, but in order to use it, I need to create a custom sink. I started creating one, but now I am stuck on creating the stream sinks. This link explains Media Sinks, but I can't find any documentation on stream sinks.

I've also looked into the Simple Communication , WavSink and another custom sink example , but the code either lacks comments or solves another problem.

Does anyone know how I can implement an UWP video calling application, or point me in the right direction?

Ps. I know this kind of question is asked more, but there are no usable answers available :(

Answer 1

Your specific implementation

The most important first step to implementing your own IMFMediaSink and IMFStreamSink classes is figuring out what you actually want to do with the IMFSample instances you'll be receiving. Once you know what kind of sink you are trying to make, then consider where you're getting the samples from, whether it's the UWP MediaCapture class, an IMFSinkWriter , an IMFTopology , or even just direct calls to your sink from client code.

If your goal is to have a two-way audio/video chat application, each instance of your app will need to capture audio and video, encode it in a compressed format, and send those samples to the other instance. MediaCapture will take care of almost all of that, except the network transmission part. Consequently, the other instance must be able to decode those audio and video samples, and present them to the user. Thus, you'll need a custom IMFMediaSink implementation, which receives IMFSample instances from MediaCapture , then transmits them over the network. You'll also need an IMFMediaSource implementation that receives samples from a network source and sends them to a presenter for rendering to the user. The presenter in this case would be an instance of MediaCapture . Each app instance will have both of the custom sources and sinks operating at the same time - one is always capturing audio and video and sending it across the network, and the other is receiving that data from the other app instance and rendering it.

One way to do this is to provide a socket interface to your transmitting media sinks and your receiving media sources. You'll need to figure out a way to handle network interruptions and cleanly resume data transmission and display on both sides. One of the easiest ways to handle this is to have a lightweight messaging layer on top of the actual video and audio sample transmission, so that you can signify a start of stream message from the sender to the receiver, which will allow the receiver to dump any queued samples and re-start its presentation clock with the first new sample that arrives, and then maintain coherent display after that. Again, the MediaCapture class will probably handle a lot of that logic for you, but you have to be careful that your sources and sinks are acting like MediaCapture expects them to.

You'll also need to establish a scheme to buffer data on each side so that small network latency doesn't cause stuttering in the audio and video. If you're playing video at 30 fps you need to display a new frame every 33 milliseconds, so that means each app instance needs to have enough data buffered to guarantee the presenter can show a frame at that interval. Audio is much the same - audio samples have durations so you need to make sure you have enough sample data for continuous playback. However, you also don't want too much buffering because you'll get a "satellite TV" effect, where the psychology of human dialog causes large gaps between each person speaking, because they are waiting for the audio transmitted by your app to stop before they start talking, and the buffering latency will magnify those gaps. Microsoft has a general discussion of buffering here , and it's useful to consider even though it specifically pertains to ASF streaming.

The last implementation related suggestion I have is to look at existing network streaming formats Media Foundation can use. Although you'll still have to implement your own sources, media sinks, and stream sinks, by writing WinRT-enabled wrappers around existing Media Foundation implementations you can avoid worrying about the vast majority of low level details. The most straightforward path I can see here is writing a wrapper class that implements IMFMediaSink , and holds an internal copy of a media sink created with MFCreateASFStreamingMediaSink . You may also need to write a WinRT-enabled IMFByteSource implementation that you can provide to your media sink, which would then be passed to the ASF streaming sink. On the source side, you'd be writing an IMFMediaSource that reads from a network byte stream and wraps an ASF file media source.

---

General overview of IMFMediaSink and IMFStreamSink

I can't give you code that will do what you want, primarily because implementing a custom media sink and set of stream sinks takes a sizable amount of work, and your needs are very specific in this case. Instead, I hope I can help you better understand the roles of media sinks and stream sinks within Media Foundation. I am far from a UWP expert, so I'll contribute as much as I can on the Media Foundation side.

From what I understand about UWP's MediaCapture class, custom sinks are responsible for interfacing with both the WinRT side and the COM / Media Foundation side, so you have to implement interfaces on both sides. This is because MediaCapture is more or less a UWP wrapper over a lot of Media Foundation technology that is abstracted away. The Simple Communication sample is actually extremely useful for this and has a lot of great starter code, particularly in the common/MediaExtensions section, which has a C++ implementation of a custom network-enabled media sink. That implementation also shows you how to interface your custom sink with WinRT so that it's usable in a UWP environment.

Below is a general discussion on the function of media and stream sinks, the general ways they are used by client code, and the general ways they can be designed and implemented.

---

Using an existing IMFMediaSink

In this section I'll go over how media sinks and stream sinks are used in practice. This will hopefully give some insights to the design decisions Microsoft made when architecting this part of Media Foundation, and help you understand how client code (even your own client code) will use your custom implementation of IMFMediaSink and IMFStreamSink .

---

Media sinks that save data

Let's look at how you'd use a specific media sink implementation within Media Foundation. We'll start with media sink you receive when you call MFCreateMPEG4MediaSink . Keep in mind that all media sinks implement the IMFMediaSink interface, no matter what type of media they represent. That function creates an IMFMediaSink that is built-in to Media Foundation, and is responsible for creating output with an MPEG4 container structure. When you create it, you have to provide an IMFByteStream into which the output MP4 data should be written. The media sink is responsible for maintaining a collection of IMFStreamSink objects, one for each input stream to the media sink. In the context of the MPEG4 media sink, there are only two possible output streams, so there exists a pre-defined number of stream sinks. This is because the MPEG4 format only allows one video stream and/or one audio stream. In this way, the MPEG4 media sink is enforcing a contract of sorts - it restricts what client code can do with it to respect the specifications of the media container it's writing. To access these sinks, you'd use IMFMediaSink::GetStreamSinkCount and IMFMediaSink::GetStreamSinkByIndex . If you have audio and video, there will be 2 sinks total with index 0 for the Video major type and index 1 for the Audio major type.

Other types of media sinks, like the one you get when calling MFCreateASFMediaSink , allow multiple audio and video streams, and require that you call IMFMediaSink::AddStreamSink for each stream you intend to write into the media sink. Looking at the documentation, you can see that you must provide a stream sink identifier (a unique DWORD or int that you want to use to reference that stream within the media sink) and an IMFMediaType that informs the media sink what kind of data you'll be sending that stream sink. In return, you receive an IMFStreamSink , and you use that stream sink to write the data for that stream.

As an aside, you can determine whether an arbitrary IMFMediaSink supports fixed or variable numbers of streams via IMFMediaSink::GetCharacteristics and checking the output flags for MEDIASINK_FIXED_STREAMS . In addition, most media sinks that store data to a byte stream will also have a flag called MEDIASINK_RATELESS which means it will consume samples as fast as you can send them, because they are all being written to a byte stream and there's no reason to wait or sync that process to a presentation clock. There will be more discussion of this in the next section.

Here's a simple worked out example, step by step. If you have an H.264 video stream and an AAC audio stream, each of them being a series of IMFSample instances, perhaps you want to save them to a file on the hard drive. Let's say you want to use the ASF container format for this. You'd create an IMFByteStream that uses a file as its storage method, and then create an ASF media sink with MFCreateASFMediaSink . Then, you'd call IMFMediaSink::AddStreamSink to add the H.264 video stream. You'd pass in a unique identifier for that stream, like 0, and the IMFMediaType that specifies things like the the media's major type (video), the media's subtype (H.264), the frame size and frame rate, and so forth. You would receive an IMFStreamSink back, and you can use that specific stream sink and its IMFStreamSink::ProcessSample method to send all your H.264 encoded video samples. Then, you'd call AddStreamSink again for the AAC audio stream, and again you'd receive back an IMFStreamSink you can use to send all the AAC encoded audio samples. The stream sinks also know who their parent IMFMediaSink is, and each stream sink works with the media sink to package the data into a single media stream, the output of which is in the IMFByteStream you created earlier. Once you have all your stream sinks set up, you'd repeatedly call IMFStreamSink::ProcessSample on each one, providing each IMFSample of the appropriate type (ie video or audio) from the source streams.

---

Media sinks that present data

Present, in this case, means "render the data as it arrives." An example of this would be the Enhanced Video Renderer (EVR), which implements the IMFMediaSink interface. The EVR is responsible for receiving IMFSample instances via its stream sinks, which you access via IMFMediaSink::AddStreamSink or IMFMediaSink::GetStreamSinkByIndex just like I discussed above. If you query the EVR media sink using IMFMediaSink::GetCharacteristics you'll see it does not provide the MEDIASINK_RATELESS flag. This is because the EVR, being a video presenter, is supposed to display the video samples it receives to the screen, and it should do so in a way that each IMFSample has its start time and duration respected. This means the EVR requires a presentation clock, via the IMFPresentationClock interface, so that it can display the video samples with the correct framerate and synchronized to the passage of time. Remember that media sinks whose job is to simply store data don't have to worry about this, because there's no need to store data at some steady or consistent rate.

There is an audio renderer similar to the EVR called the Streaming Audio Renderer (SAR) and it works in a similar way. As a thought experiment, consider the basic architecture if you wrote your own media sink that wrapped both the EVR and SAR, so you could connect video and audio streams to your custom sink, and the custom stream sink implementations would feed the examples to the appropriate renderer based on the media type of each stream sink. That way you could just create one media sink instead of worrying about the EVR and SAR as separate media sinks.

---

Implementing your own IMFMediaSink and IMFStreamSink

At this point, I hope you can see the implementation of IMFStreamSink is entirely dependent on the implementation of IMFMediaSink - in other words, an MPEG4 media sink is going to need its own specific implementation of IMFStreamSink , because those stream sinks need to package the raw media data inside MPEG4 containers, need to build an index of samples and timestamps to embed in the MPEG4 file, and so forth.

---

Writing your own version of the MPEG4 sink

Given what has been presented above, consider how you would structure the actual classes that hide behind those interfaces if you were writing your own version of an MPEG4 sink. You'd need an implementation of IMFMediaSink , a way to create an instance of that implementation, and then you'd need to have one or more implementations of IMFStreamSink that are directly dependent on your implementation of IMFMediaSink .

Here's one way to structure your implementation. You'd start with a class called CMPEG4MediaSinkImpl which implements IMFMediaSink , and perhaps you'd have a static C-style function to create instances of that class, just like MFCreateMPEG4MediaSink . You only support one video and one audio stream, and the media types for each are provided to the non-public constructor of your CMPEG4MediaSinkImpl class. Let's say you wanted to have different stream sinks based on the media's major type, ie audio or video, perhaps because you want separate counts of how many audio and video samples you received. The constructor would check each of the media types, and for the video stream it would create an instance of CMPEG4StreamSinkVideoImpl , and for audio it would create an instance of CMPEG4StreamSinkAudioImpl . Each of those class implementations would implement IMFStreamSink and would be responsible for receiving the individual IMFSample instances in their IMFStreamSink::ProcessSample implementations. Perhaps in those implementations, each stream sink would call CMPEG4MediaSinkImpl::WriteSampleBlock that would create the appropriate MPEG4 data structure to wrap the sample data, as well as document the sample time and duration in an MPEG4 structure, and then embed that in the output IMFByteStream .