Cant display my video stream from a python client in the HTML client through a flask-socketIO server

Question

I would like to display my video stream from my jetson nano in a HTML page but it seems i cant make it work as i would like. I have a flask-socketIO server in python, I have my python client which send each frame to this server ( it is where my webcam is attached) and then the server broadcast the data to the HTML client render by the flask-socketIO In the browser console, i can see the data but the image is not displayed.

I also tried to display back the data directly from the server without broadcasting to the HTML but it was the same problem. with openCV i could not use imshow to display the image but i could save the frame after to the disk with imwrite.

Here is the code from my python client, basically, it retrieves the frame, convert it it as byte and then encode to base 64 and emit with socketIO

while(stream):
     ret, frame = cap.read()
     _, buff = cv2.imencode('.jpeg', frame)
     jpg_as_text = base64.b64encode(buff)
     socketIO.emit('my image data', {'image': True, 'buff': jpg_as_text})

Here is the server side. It just broadcast the data from python client to the HTML client:

@socketio.on('my image data')
def broadcast_img(data):
    emit('imageStream', data, broadcast=True)

And here is the HTML client which is render by the server:

var ctx = document.getElementById("myStream").getContext('2d');
var socket = io();
socket.on('imageStream', function(data) {
    console.log(data)
    if(data.image) {
      var img = new Image();
      img.src = 'data:image/jpeg;base64,' + data.buff;
      ctx.drawImage(img, 0, 0);
    }
});

When i open the browser and check the console, i can see the value for buff. It starts with /9j/4AAQ....... which i assume is the encoded image

Does someone have an idea why i cant display the image?

Thanks in advance

Answer 1

data.buff is an ArrayBuffer that you have to decode as string:

const decoder = new TextDecoder();
img.src = 'data:image/jpeg;base64,' + decoder.decode(data.buff);

Besides, you don't have to Base64 decode image data. You may just emit binary data

_, buff = cv2.imencode('.jpeg', frame)
socketIO.emit('my image data', {'image': True, 'buff': BytesIO(buff).getvalue()})

and use it on the client like this:

const arrayBufferView = new Uint8Array(data.buff);
const blob = new Blob([arrayBufferView], {type: 'image/jpeg'});
img.src = URL.createObjectURL(blob);