通过 UNIX 插座在 Python 和 Rust 之间发送和接收视频

Question

The final requirement is to create a system which can stream video footage via a UNIX IPC socket from Python to Rust. The Python script has exclusive access to the camera/video. I'm pretty new to Rust.

I have tried an approach wherein the control flows as follows:

• In Python, as the video flows in, strip each frame out of it to convert it into a NumPy array
• Then, convert it into a string. Then to bytes.
• Send it over a UNIX IPC socket.
• At the receiving end, convert it back into string.
• Hopefully parse it to get a usable array and make an image from it.

Sender:

import socket, os, cv2, numpy
# print all the string without truncating
numpy.set_printoptions(threshold=numpy.inf)

# declare the camera socket and unlink path if already in use
camera_socket_path = '/home/user/exps/test.sock'
try:
    os.unlink(camera_socket_path)
except OSError:
    pass

# create and listen on the socket
camera_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
camera_socket.bind(camera_socket_path)
camera_socket.listen()

conn, addr = camera_socket.accept()

vidcap = cv2.VideoCapture('example.mp4')
success, image = vidcap.read()
print(str(image))
while success:
    conn.send(bytes(str(image), 'utf-8'))      
    success, image = vidcap.read()
    break # after sending a frame, for testing

conn.close()

Receiver:

use std::os::unix::net::UnixStream;
use std::io::prelude::*;

use image::RgbImage;
use ndarray::Array3;

fn array_to_image(arr: Array3<u8>) -> RgbImage {
    assert!(arr.is_standard_layout());

    let (height, width, _) = arr.dim();
    let raw = arr.into_raw_vec();

    RgbImage::from_raw(width as u32, height as u32, raw)
        .expect("container should have the right size for the image dimensions")
}

fn main() {
    // create a standard UNIX IPX socket stream
    let mut stream = UnixStream::connect("/home/user/exps/test.sock").unwrap();

    loop {
        // 2074139 is the length of the string printed at sender :D
        // I thought it might work out but it didn't obviously
        // the docs suggest an exponential of 2. default was 1024
        let mut buf = [0; 2074139];
        let count = stream.read(&mut buf).unwrap();
        let response = String::from_utf8(buf[..2074139].to_vec()).unwrap();
        // write a parser function to convert the string to Array3 iterating through line()
        // let pic = array_to_image(convert(response));
        println!("{}", response);
        break; // get a single frame, for testing
    }
}

Both the strings look nothing like each other. In the final version, I hope to create a stream using which I can continuously do this for incoming video stream from a camera, with something like cv2.VideoCapture() . What is the list of stuff to be done to achieve it?

Answer 1

An mp4 file consists of arbitrary byte sequences, not utf8 strings. So you should send the raw bytes and receive into a Vec<u8> or similar data type.

There is no reason to mangle things through a string encodings, unix sockets support arbitrary bytes.

For debugging purposes you can print out the bytes in hex format, but you shouldn't send it as such since that too would be wasteful.