从传入的原始图像数据更新 HTML 页面上的视频帧

Question

I have raw image data (1000 x 1000 pixels x 3 bytes per pixel) in Python, that I need to send to a HTML page in realtime, at 20 frames per second (this is 57 MB of data per second!).

I already tried the multipart/x-mixed-replace method (as seen in Sending RGB image data from Python numpy to a browser HTML page ), with various encoding: BMP, PNG, JPG. It is quite intensive for the CPU, so I'm trying alternatives.

I am now getting the raw image data directly in JavaScript with binary XHR HTTP requests.

Question: How to (CPU-efficiently) decode binary RGB data from dozens of binary XHR HTTP requests into a <video> or <img> or <canvas> on a HTML page, with Javascript?

oReq = new XMLHttpRequest();
oReq.open("GET", "/imagedata", true);
oReq.responseType = "arraybuffer";
oReq.onload = function (oEvent) {
  var arrayBuffer = oReq.response;
  if (arrayBuffer) {
    var byteArray = new Uint8Array(arrayBuffer);
    // update displayed image
  }
};
oReq.send(null);

---

Edit: The method given in @VC.One's comment: var byteArray = new Uint8ClampedArray(arrayBuffer); var imgData = new ImageData(byteArray, 1000, 1000); var ctx = myCanvas.getContext('2d'); ctx.putImageData(imgData, 0, 0); var byteArray = new Uint8ClampedArray(arrayBuffer); var imgData = new ImageData(byteArray, 1000, 1000); var ctx = myCanvas.getContext('2d'); ctx.putImageData(imgData, 0, 0); works and is lighter for the CPU: 5%-6% for the server sending the data vs. 8%-20% with BMP/PNG/JPG encoding . But Chromium now has two processes in parallel for this task, each of them ~ 15% CPU . So the total performance is not much better. Do you see other potential alternatives to efficiently send raw image data from a Python or C++ HTTP server to Chromium?

Also new ImageData(...) requires a 1000x1000x4 bytes array for R, G, B, A. This requires that I send alpha channel that I don't need; maybe there's a way with ImageData to only pass a RGB (nxnx3 bytes) array?

---

Edit 2: the real bottleneck is the XHR HTTP requests between my process #1 and process #2 (Chrome) on the same computer for up to 100 MB/sec. Is there a more direct inter process communication possible between process #1 and Chrome? (some sort of direct memory access?)

See Chrome + another process: interprocess communication faster than HTTP / XHR requests?

Answer 1

There is another way, but it will take a lot of new material. First, use requests on python to send your info to a page, say /internals/port by doing something like this:

import requests

url = 'http://random.com/internals/port'
query = {'field': value}
res = requests.post(url, data=query)

then, use a fetch() from your website in js. for example:

fetch('/internals/port').then(res=>{
res.text().then(text=>{
//here text is your data. 
// you can also use an await fetch statement so things aren't wrapped up in the two functions. 
})
})

Try instead to send your info in 64bit since you can use data:image/png;64bit as its own 64bit parser:

data:image/png;64bit,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

if you insist on using binary, then try using a filereader to parse it into 64bit,

var reader = new FileReader(
  reader.onload = (function(self) {
    return function(e) {
      document.getElementById("img").src = e.target.result;
    }
  })(this);
  reader.readAsDataURL(new Blob([r]));

it will take a bit more time but still worth it.

you can just use an <img> tag in your html and set the url to a data:image/png;base64 data url that you generate. Still, coming back on your comment on the ImageData() class you can just set the alpha to 1, or 100%, since it is the opacity, so just incluce a 1 at the end of the array when you feed it into new ImageData() . Good luck on your project, I hope you successfully find a way to do what you want to, whether it be with my answer or with another. I believe that I have answered all your questions, and as a final suggestion, try changing your python backend to node.js, and use express requests to post images that can be accessed in pretty much the same way as using requests in python. It would be much more efficient and much faster.