How can I initialise an RGBA cv::Mat from a byte array?

Question

I'm sending images over the network (from Python) and want to create OpenCV Mat s from them on the receiving end (in C++).

They are created like this:

image = self.camera.capture_image()   # np.array of dtype np.uint8
h, w, c = image.shape   # 4 channels
image = np.transpose(image, (2, 0, 1)) # transpose because channels come first in OpenCV (?)
image = np.ascontiguousarray(image, dtype='>B')  # big-endian bytes
bytess = image.tobytes(order='C')

After this, I should have an array where the 3 dimensions are flattened such that individual rows are appended together for each channel and then the channels are appended to form the final byte buffer. I have verified that my understanding is correct and the following holds

bytess[channel*height*width + i*wwidth + j] == image[channel, i, j]

[I think the above part is actually unimportant, because if it's incorrect, I will get an incorrectly displayed image, but at least I would have an image, which is one step further than I am now.]

Now on the other side I am trying to do this:

char* pixel_data = … // retrieve array of bytes from message
// assume height, width and channels are known
const int sizes[3] = {channels, width, height};
const size_t steps[3] = {(size_t)height * (size_t)width, (size_t)height};
cv::Mat image(3, sizes, CV_8UC1, pixel_data, steps);

So, I create a Matrix with three dimensions where the element type is byte . I am not so sure I'm determining the steps correctly, but I think it matches the documentation .

But running this just crashes with

error: (-5:Bad argument) Unknown array type in function 'cvarrToMat'

What is the correct way to serialise an RGBA (or BGRA for OpenCV) image to a byte buffer and create a cv::Mat from it with the C++ API?

Answer 1

I have one solution, which circumvents the problem. This line here:

cv::Mat image(3, sizes, CV_8UC1, pixel_data, steps);

makes the assumption that I can pass sizes of three dimensions with individual bytes, but I could not make this work.

Instead using a different constructor

cv::Mat image(height, width CV_8UC4, pixel_data);

I can treat the image as two-dimensional but with a vector-datatype (4 bytes element size instead of scalar bytes). If the pixel_data pointer is in the correct layout, this works.

The correct layout is not really explicitly documented, but can be deduced from one of the official tutorials

So the data is stored such that one row comes after the other and each element of a row is split into n_channels elements. Using a data type such as CV_8UC4 makes the matrix read 4 bytes at each position in the raw data array, and advance the pointer 4 bytes.

So in this case, I just have to rearrange the numpy array into the appropriate sequence: append rows together, but interleave the channels. I did this like so, but I hope there's a way without looping.

def array_to_cv_bytes(ary):
    assert ary.ndim == 3, 'Array must have 3 dimensions'
    h, w, c = ary.shape
    ary = ary[..., (2, 1, 0, 3)]
    output = np.empty(h * c * w, dtype=np.uint8)
    for channeld_idx in range(c):
        output[channeld_idx::c] = ary[..., channeld_idx].reshape(h*w)
    return output.tobytes(order='C')