Is there any way to resume live h264 playback (client rejoin) using the h.264 codec?

Question

Currently, I'm having a bit of an issue with the MediaSource API for resuming live h264 playback. My server-side code will keep the first packet from FFmpeg, then dispatch it to clients. However, this works well but raises an issue.

When I restart the stream and it dispatches to the client(s), it goes as expected and this is the result I expect when a client disconnects then has to reconnect to the stream, is the aforementioned result

However, if I were to rejoin the stream, I get this as the result

Also, FFMpeg data is being sent to the client in the picture above, it's just not rendering it for some reason.

Here's my function for playing audio/video frames that I get from the server.

private _playFrame(type: 0 | 1) {
    const src = type === 0 ? this.audioSource : this.videoSource;
    if (!src || src.updating) return;
    const queue = type === 0 ? this.audioFrameQueue : this.videoFrameQueue;
    src.appendBuffer(queue.shift());
    if (this.video.src && this.video.paused) this.video.play().then(() => null);
}

Answer 1

Fragmented.mp4 data streams -- the type used for live playback -- have a prologue describing their media before the compressed media itself.

The prologue containing the media metadata is a chunk of data -- an "atom" in the lingo of mp4 -- named 'moov' . One of its subatoms, 'avcC' , contains so-called codec-private data for H.264 video streams. If you ask a decoder to process H.264 without giving it the codec-private data, it cannot interpret the H.264 and so skips over it. This is true of any decoder, including the ones embedded in browser or desktop media player packages.

The 'moov' prologue is probably in the first packet from ffmpeg (although you'd need to examine that data with a tool like mp4dump to be sure).

So, to join a live stream a viewer must receive the prologue data and then the live data.