将 RGB 图像数据从 Python numpy 发送到浏览器 HTML 页面

Question

I need to send realtime image RGB data (in Numpy format) to a HTML page in a browser (web-based GUI), through HTTP. The following code works with the well-known multipart/x-mixed-replace trick: run this and access http://127.0.0.1:5000/video_feed : you will see a video in the browser.

from flask import Flask, render_template, Response
import numpy as np, cv2
app = Flask('')
def gen_frames():  
    while True:
        img = np.random.randint(0, 255, size=(1000, 1000, 3))
        ret, buf = cv2.imencode('.jpg', img)
        frame = buf.tobytes()
        yield (b'--frame\r\nContent-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')
@app.route('/video_feed')
def video_feed():
    return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')
app.run()

However, according to my benchmark, the real performance bottleneck is the cv2.imencode('.jpg', img) .

In my real application, if I just generate the image, the CPU is ~ 1% for Python. When I imencode(...) , the CPU jumps to 25%, and 15% for Chrome.

I also tried with PNG format but it's similar.

Question: how to efficiently send RGB image data from a numpy array (example: 1000 x 1000 pixels x 3 colors because of RGB) to a browser HTML page?

(Without compression/decompression it might be better, but how?)

Here is the benchmark

          FPS       CPU PYTHON      CPU CHROME
PNG       10.8      20 %            10 %
JPG       14        23 %            12 %
JPG       10.7      16 %            10 %           (with time.sleep to match PNG 10.8 fps)
BMP       19        17 %            23 %
BMP       10.8      8 %             12 %           (with time.sleep to match PNG 10.8 fps)

Answer 1

Try using the PILLOW module instead and see if that improves performance. My benchmark shows that each iteration of the gen_frames() generator function based on PILLOW requires less than half the CPU of the CV2 version.

from flask import Flask, render_template, Response
from PIL import Image
import numpy as np
from io import BytesIO

app = Flask('')

def gen_frames():
    while True:
        img = np.random.randint(0, 255, size=(1000, 1000, 3), dtype=np.uint8)
        rgb_image = Image.fromarray(img, 'RGB')
        buf = BytesIO()
        rgb_image.save(buf, 'JPEG')
        frame = buf.getbuffer()
        yield (b'--frame\r\nContent-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

@app.route('/video_feed')
def video_feed():
    return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')

app.run()

Answer 2

According to the docs you could check if the default optimization is turned on:

Default Optimization in OpenCV

Many of the OpenCV functions are optimized using SSE2, AVX, etc. It contains the unoptimized code also. So if our system support these features, we should exploit them (almost all modern day processors support them). It is enabled by default while compiling. So OpenCV runs the optimized code if it is enabled, otherwise it runs the unoptimized code. You can use cv.useOptimized() to check if it is enabled/disabled and cv.setUseOptimized() to enable/disable it.

So try this:

In [5]: cv.useOptimized()         # check for optimization
Out[5]: False
In [7]: cv.setUseOptimized(True)  # turn it on if not already turned on

Answer 3

As it stands AVI or MP4 compression would be good quality even for movies, but the compression itself takes too much CPU time to perform it on live data.

If some arbitrary protocol/format were created, one would not just have to program the server but also the client to consume this protocol/format. Therefore still some standard solution should be preferred.

I believe you can find a compromise between compression and CPU load in video conferencing systems, where the live camera data needs to be compressed and streamed via the network. With that in mind I believe here are sources of information that can help pursuing the topic:

• https://www.reachcambridge.com/wp-content/uploads/wp_videocompression_33085_en_0809_lo.pdf
• https://cs.haifa.ac.il/~nimrod/Compression/Video/V3h261-2005.pdf
• Open Source Video Conferencing software like Jitsi Meet or Apache OpenMeetings .

Answer 4

Maybe you can try encoding it as base64 for compression of the video/image and then send it to the browser with the base64 mime type egdata:image/jpeg;base64.