从 HTML Canvas 捕获图像的最快方法

Question

Here's my code to capture an image from a Canvas playing video:

    let drawImage = function(time) {
        prevCtx.drawImage(videoPlayer, 0, 0, w, h);
        requestAnimationFrame(drawImage);
    }
    requestAnimationFrame(drawImage);

    let currIndex = 0;
    setInterval(function () {
        if(currIndex === 30) {
            currIndex = 0;
            console.log("Finishing video...");
            videoWorker.postMessage({action : "finish"});
        } else {
            console.log("Adding frame...");
            // w/o this `toDataURL` this loop runs at 30 cycle / second
            // so that means, this is the hot-spot and needs optimization:
            const base64img = preview.toDataURL(mimeType, 0.9);
            videoWorker.postMessage({ action: "addFrame", data: base64img});
            currIndex++;
        }
    }, 1000 / 30)

The goal is at each 30 frames (which should be at 1 second) it would trigger to transcode the frames added.

The problem here is that the preview.toDataURL(mimeType, 0.9); adds at least 1 second, without it the log shows the currIndex === 30 gets triggered every second. What would be the best approach to be able to capture at least about 30 FPS image. What is the fastest way to capture image from a HTML Canvas that it will not be the bottleneck of real-time video transcoding process?

Answer 1

You should probably revise your project, because saving the whole video as still images will blow out the memory of most devices in no time. Instead have a look at MediaStreams and MediaRecorder APIs, which are able to do the transcoding and compression in real time. You can request a MediaStream from a canvas through its captureStream() method.

---

The fastest is probably to send an ImageBitmap to your Worker thread, these are really fast to generate from a canvas (simple copy of the pixel buffer), and can be transferred to your worker script, from where you should be able to draw it on a an OffscreenCanvas.

Main drawback: it's currently only supported in latest Chrome and Firefox (through webgl), and this can't be polyfilled...

main.js

else {
  console.log("Adding frame...");
  const bitmap = await createImageBitmap(preview);
  videoWorker.postMessage({ action: "addFrame", data: bitmap }, [bitmap]);
  currIndex++;
}

worker.js

const canvas = new OffscreenCanvas(width,height);
const ctx = canvas.getContext('2d'); // Chrome only
onmessage = async (evt) => {
  // ...
  ctx.drawImage( evt.data.data, 0, 0 );
  const image = await canvas.convertToBlob();
  storeImage(image);
};

---

An other option is to transfer an ImageData data. Not as fast as an ImageBitmap, it still has the advantage of not stopping your main thread with the compression part and since it can be transferred, the message to the Worker isn't computation heavy either.
If you go this road, you may want to compress the data using something like pako (which uses the compression algorithm used by PNG images) from your Worker thread.

main.js

else {
  console.log("Adding frame...");
  const img_data = prevCtx.getImageData(0,0,width,height);
  videoWorker.postMessage({ action: "addFrame", data: img_data }, [img_data.data]);
  currIndex++;
}

worker.js

onmessage = (evt) => {
 // ...
 const image = pako.deflate(evt.data.data); // compress to store
 storeImage(image);
};