在 Raspberry Pi 上保存图像流的最快方法

Question

We have a camera connected to a Raspberry Pi 4 via USB3. The camera can only provide RAW images [2056x1542x3] which we can read at about 30 FPS. On the Raspberry Pi 4 we need to save these images to disk - due to space and write speed to the SD card it is not feasible to save the RAW images (10 MB/piece) at any rate really. Instead we want to compress these images and then save them as fast as possible.

Our current solution looks similar to the below snippet:

def save_image(frame,filename):
    cv2.imwrite(filename,frame)

...

(ret, frame) = cam.get_next_frame()
if ret == IS_SUCCESS:
    timestamp = get_timestamp()
    filename = conf["cv_image_folder"] + timestamp + ".jpg"
    save_thread = threading.Thread(target=save_image, args=(frame,filename,))
    save_thread.start()

OpenCV is compiled with libjpeg-turbo and all possible HW-flags for accelerated computing. At about 5 or 6 FPS the Raspberry Pi 4 uses about 100% of all 4 cores. The same goes for a non-threaded configuration. We manually set the framerate on the camera and monitor the number of threads spawned (which is about 3-4 concurrent threads at 5-6 FPS). We choose JPEG (even though its lossy) since PNG or TIFF compression takes longer to compute.

Is there any way to improve this?

Answer 1

I think Dan's discovery that you can get the Bayer image is going to prove the solution to your problem, however I did some work on the concept of using Broadcom's MMAL "Mulitmedia Abstraction Layer" to do JPEG encoding on the VideoCore GPU to lessen the CPU load and wanted to place it here. It may prove useful to you in combination with Dan's suggestion and/or to other folk seeking to do similar things.

As a result of running on the GPU, it can encode your (rather large) images to disk at around 4 frames per second whilst using almost no CPU . I observed 3 CPU cores were idle and the fourth core hovered on and off around 10-15% usage.

The code just takes a single image called "image.jpg" and resizes it to the dimensions you are using and then repeatedly encodes that and sends it to disk 100 times.

#!/usr/bin/env python3

import io
import numpy as np
from picamera import mmal, mmalobj as mo
from PIL import Image
from threading import Event

# Globals
w, h = 2048, 1536      # These seem to need to be multiples of 64, or maybe 16
EncoderAvail = Event()
EncoderAvail.set()

# Set up MMAL-based JPEG encoding
encoder = mo.MMALImageEncoder()
encoder.inputs
encoder.inputs[0].format = mmal.MMAL_ENCODING_RGB24
encoder.inputs[0].framesize = (w, h)
encoder.inputs[0].commit()
encoder.outputs[0].copy_from(encoder.inputs[0])
encoder.outputs[0].format = mmal.MMAL_ENCODING_JPEG
encoder.outputs[0].commit()
encoder.outputs[0].params[mmal.MMAL_PARAMETER_JPEG_Q_FACTOR] = 90

def image_callback(port, buf):
    with open(f"frame-{frame}.jpg", 'wb') as jpeg:
        jpeg.write(buf.data)
        EncoderAvail.set()
    #return bool(buf.flags & mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END)
    return True

encoder.outputs[0].enable(image_callback)
encoder.inputs[0].enable(lambda port, buf: True)

# Encode the same image repeatedly as we are measuring the encode time not acquire time
im = Image.open('image.jpg').resize((w,h))
rgb_data = np.array(im)

for frame in range(100):
    print(f"Frame: {frame}")

    # Make red bar across top that gets longer with each frame to be sure we are getting them all
    rgb_data[0:10,:frame,0] = 255

    EncoderAvail.wait()
    buf = encoder.inputs[0].get_buffer()
    buf.data = rgb_data[:]
    EncoderAvail.clear()
    encoder.inputs[0].send_buffer(buf)

encoder.outputs[0].disable()
encoder.inputs[0].disable()

The code is heavily based on the Picamera MMAL stuff - here . The code is probably not optimal - for example, I do not know how to increase the number of buffers to 2 which would enable double-buffering and potentially make a big difference. I am also not sure how to make it more event-driven as it is rather synchronous at the moment.

As I said, you can change it to encode Bayer images to JPEG, by changing mmal.MMAL_ENCODING_RGB24 so you may be able to still use it.

If anyone knows more about the MMAL stuff and is able to improve it, please feel free to take my code, improve it and add it as a new answer and let me know so we can all learn.

I noticed that it doesn't currently seem to honour the JPEG Quality setting either, so if anyone can improve that, please let me know.

Keywords : Raspberry Pi, GPU, JPEG encoding, JPEG encoder, MMAL, Videocore GPU, Broadcom, Multimedia Abstraction Layer, Raspi.