Real Time video Processing Design issue

Question

I am a newbie in the field and don't know if i am doing this right. So I need help with architectural and design issue, just to make sure if I am moving in right direction.

I am trying to process a real time video capturing from the client's camera. The captured video stream is converted into frames and send to the server for processing. After processing, server is sending it back to client's screen.

My backend code is written in Python and I am using SocketIo to send the frames from frontend to backend. You can have a look at this design to get a better idea about what's happening - image

1. My server(app.py) will be running in backend and client will be accessing index.html
2. SocketIo connection will get establish and video stream captured using webcam will be send to server frames by frames.
3. These frames will be then processed at the backend and emit back to the client.
4. Processed frames coming form the server can be shown in img tag.

So, my problem with this architecture is the lag. There is a noticeable delay even after limiting fps. From client to server then processing and back to client, each frame is take on an average 150 ms and is not giving a real time response. I have tried my image processing code separately with no socket programming on local and it is more near to real time (it is taking 80 ms for one frame).

But I fail to understand here that how can I minimize lagging with this design or improvise it? Is there any better way to do this task to get more real time results?

Thanks in advance!

Answer 1

Python is a great language for rapid prototyping Computer Vision applications. However, it is not recommended for production code because of its performance. Convert the code to C++ and you'll see significant improvement in runtime performance.

SocketIo its not the best option since it uses a TCP connection. You'll notice better performance from a library that transfers data through UDP. You might loose a few pixels here and there, but it's usually a good trade-off for streaming videos.

Here's a few more thoughts on SocketIo and TCP vs UDP. .