Sending frames using sockets in Python

Question

I'm trying to get frames from client, send it to server and from there write it into a video. But I keep failing in sending part, getting TypeError: Expected Ptr<cv::UMat> for argument '%s' error in out.write(frame) .

I've also tried using pickle.dumps(frame) and then loading it in server side but it keeps getting truncated.

Server:

import numpy as np 
import cv2, socket

fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter("output.avi", fourcc, 19.0, (1366, 768))

s = socket.socket()
host = socket.gethostname()
port = 8080
s.bind((host,port))
s.listen(1)
print(host)
print("Waiting for any incoming connections ... ")
conn, addr = s.accept()
print(addr, "Has connected to the server")

while True:
    frame = conn.recv(1024)
    # write frame to video writer
    out.write(frame)

    if cv2.waitKey(1) == 27:
        break

out.release()
cv2.destroyAllWindows()

Client:

import numpy as np 
import cv2, socket
from PIL import ImageGrab

s = socket.socket()
host = input(str("Please enter the host address of the sender : "))
port = 8080
s.connect((host,port))
print("Connected ... ")


while True:
    img = ImageGrab.grab()
    img_np = np.array(img)
    frame = img_np
    s.send(frame)

Apparently in server, frame becomes <class 'bytes'> . So, I'm trying to find any way to fix this, including somehow converting bytes back into ndarray, or finding any other workaround.

Thanks.

Answer 1

Lets separate your question into two parts:

1. How to send data over a socket?
   You are using a socket with 1024 bytes buffer which means that in every iteration you get 1024 bytes data at maximum.
   What you should do when working in low level networking, is to put a unique end identifier token in the end of the frame and iterate in the server side with .recv() until you reached it. Another option is to send the length of your message and count the received bytes. This way, you know when you have a complete frame, then you can break the while loop, convert it to numpy array and .write() it.

2. How to pass numpy array over network?

   • You can pickle it and transfer the bytes into a io.BytesIO stream. Then load the stream with np.load() function.
   • You can also serialize the frame pixels as array of your pixel type, read them from the socket into io.BytesIO, then read them into numpy with np.fromfile(..., dtype= ...)