Smooth variable frame-rate (video)

Question

I have a motion-jpeg decoder that I wrote for viewing ip-camera streams. It works great, I can easily achieve over 30 fps with multiple devices. My problem is that these frames can come in bursts over the network.

What I'm trying to achieve is a formula for averaging the time between frames to get a more steady playback. Currently, my frames are put into a ConcurrentQueue from the network thread, and the most recent frame is displayed on the UI thread. Below is my current code for smoothing the video stream, but it doesn't work as I planned...

PlaybackFrame ) Class holding BitmapImage -> " Image "
base.getFrame() ) retrieve PlaybackFrame from ConcurrentQueue

    private const int MAX_FRAME_DELAY = 500;
    private const float RATE_FACTOR = 0.1f;

    private long last_frame_time;
    private long last_update_time;
    private float rate;

    //=============================


    public override bool Get(out BitmapImage image) {
            PlaybackFrame f = null;
            if (base.getFrame(out f)) {
                long now = Environment.TickCount;
                if (last_frame_time > 0) {
                    // Get # of frames in buffer
                    int count = getCount();
                    //
                    // Get duration since last update & last frame displayed
                    int update_duration = (int)(now - last_update_time);
                    int frame_duration = (int)(now - last_frame_time);
                    //
                    // estimated delay based on current frame-rate
                    float target_rate = 0;
                    if (count > 0) target_rate = update_duration / (float)count;
                    //
                    // offset actual delay/rate by current value
                    last_update_time = now;
                    rate = lerp(rate, target_rate, RATE_FACTOR);
                    //
                    // [backup] if duration exceeds 0.5 seconds, display next frame
                    if (frame_duration >= MAX_FRAME_DELAY) {
                        image = f.Image;
                        last_frame_time = now;
                        return true;
                    }
                    //
                    // if duration exceeds delay, display image
                    if (frame_duration > rate) {
                        image = f.Image;
                        last_frame_time = now;
                        return true;
                    } else {
                        // too soon, wait...
                        image = null;
                        return false;
                    }
                } else {
                    // first image, display
                    last_frame_time = now;
                    image = f.Image;
                    return true;
                }
            } else {
                // no image available
                image = null;
                return false;
            }
        }

        private float lerp(float a, float b, float f) {
            return a*(1f - f) + b*f;
        }

Answer 1

Found my mistake. I was computing my time measurements within the base.getFrame() method, which only executes if a frame is available. By moving the measurements outside of this block, they are updated on every render event, creating a smooth time-step based on the number of frames available within the buffer.

• Still needs to be cleaned up, but working great...

   private const int MAX_FRAME_DELAY = 500; private const float RATE_FACTOR = 0.01f; private long last_frame_time; private long last_update_time; private float rate; //============================= public override bool Get(out BitmapImage image) { PlaybackFrame f = null; long now = Environment.TickCount; if (last_frame_time > 0) { int count = getCount(); // int update_duration = (int)(now - last_update_time); int frame_duration = (int)(now - last_frame_time); // float target_rate = 0; if (count > 0) target_rate = update_duration / (float)count; // last_update_time = now; rate = lerp(rate, target_rate, RATE_FACTOR); // if (frame_duration >= MAX_FRAME_DELAY) { if (getFrame(out f)) { rate = MAX_FRAME_DELAY; last_frame_time = now; image = f.Image; return true; } else { image = null; return false; } } // if (frame_duration > rate) { if (getFrame(out f)) { last_frame_time = now; image = f.Image; return true; } else { image = null; return false; } } else { image = null; return false; } } else { if (getFrame(out f)) { last_frame_time = now; image = f.Image; return true; } else { image = null; return false; } } } private float lerp(float a, float b, float f) { return a*(1f - f) + b*f; }