ffmpeg: 1 frame manipulation on demand

Question

Hello folks & friends of SO!

We need to create a small application to take a random frame from a video, and apply some modifications to it, in real time.

This has to be done in real time, and in order to do that, we wouldn't mind to create some initial delay, before the video starts streaming. So let's assume we have a total of 6 seconds of preloading time, in order to:

1) extract a keyframe.
2) create an image with it.
3) manipulate the image.
4) replace the original keyframe with the new one.

The point (3) of the previous algorithm is already solved using GD as a PHP module. However we would need to find a solution for the rest of the procedure in order to accomplish this.

Please if anyone is available to help us with this, feel free to contact me.

Thanks in advance; Chris C. Russo

Answer 1

This is a pretty complicated problem that cannot be solved by simply replacing the keyframe with a modified image. Following problems (at least) should be addressed for this to work:

• Re-encoding all the frames that depend on the keyframe. Otherwise, the video would be corrupt.
• File format considerations - The modified keyframe and subsequent frames are likely to have a different size after encoding (compared to the original encoded data). Storing them in the output file is not easy
• Timestamp - The timestamp information for these modified frames must be preserved, else there would be audio-video sync issues in the output file.

Given these issues, my recommendation is to perform the image manipulation in the raw video frame, and encode the stream afterwards. Please let me know if I can help clarify.