How do you run two programs in python simultaneously at different frame rates?

Question

I have been experimenting with OpenCV in python and I want to make a game with it in pygame. But it is way too slow, OpenCV is running at 8 fps but I want the pygame game to be running at 60 fps . I don't need the tracking to be fast but I want pygame to be running at a normal frame rate. I have tried to use the built-in sockets in python but it would make both programs run at 8 fps. Then I tried to stop the socket from pausing at waiting for data but changed nothing. Thanks.

pose.py

import cv2
import mediapipe as mp
import time
import numpy as np
cap = cv2.VideoCapture(0)

mpHands = mp.solutions.hands
hands = mpHands.Hands(static_image_mode=False,
                      max_num_hands=8,
                      min_detection_confidence=0.5,
                      min_tracking_confidence=0.5)
mpDraw = mp.solutions.drawing_utils

pTime = 0
cTime = 0

while True:
    success, img = cap.read()
    imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    results = hands.process(imgRGB)
    #print(results.multi_hand_landmarks)

    if results.multi_hand_landmarks:
        handId = 0
        for handLms in results.multi_hand_landmarks:
            avX = avY = l = 0
            for id, lm in enumerate(handLms.landmark):
                #print(id,lm)
                h, w, c = img.shape
                l += 1
                cx, cy = int(lm.x *w), int(lm.y*h)
                avX += cx
                avY += cy
                #if id ==0:
                cv2.circle(img, (cx,cy), 3, (255,0,2555), cv2.FILLED)
            avX = avX / l
            avY = avY / l

            mpDraw.draw_landmarks(img, handLms, mpHands.HAND_CONNECTIONS)
            handId += 1


    cTime = time.time()
    fps = 1/(cTime-pTime)
    pTime = cTime

    cv2.putText(img,str(int(fps)), (10,70), cv2.FONT_HERSHEY_PLAIN, 3, (255,0,255), 3)

    #cv2.imshow("Image", img)
    cv2.waitKey(1)

poseGame.py

import pygame
pygame.init()

screen = pygame.display.set_mode((960, 540))
clock = pygame.time.Clock()
while True:
    screen.fill((0, 0, 0))

    # ...
    
    clock.tick(60)
    pygame.display.update()

Answer 1

you have three options.

first option is to have the game or the app running as the main python application and the other one running as a child process using the multiprocessing module and use a manager.queue for communication between the two processes, where the game will check if there is something in the queue on each frame (and pop the message if it finds one).

second (harder) option is to use sockets in non-blocking mode using socket.setblocking , so the game will not block until the other app finishes the transmission, but you will need to incrementally update an internal "message buffer" on each frame of the game as the messages will be received on multiple frames, the advantage of this approach is that the game and the commanding application will be two separate applications with no shared code, so the communication doesn't even have to be within python.

third option is to employ threading with blocking sockets where a child thread will be monitoring the received messages from the other application and putting it on a deque or the main thread to read.

one disadvantage of both approaches is that the application has to constantly send messages to the game, as both applications will be running at different rates, therefore the game needs to save the last message and keep using it until it receives the next message.

Answer 2

You need threads.

Do not mess around with sockets for this, it's needlessly complicated. You also don't necessarily need multi processing . Threads are just fine. All the heavy lifting (OpenCV, mediapipe, ...) will release the GIL. If you don't know what that means, it's okay. I'm saying that for those that think threading isn't okay to use.

You need a thread for the mediapipe stuff and another thread for reading the camera. And you can do your game logic in the main thread (the one you didn't start yourself).

The thread for the camera reading is required because if you don't read from the camera as quickly as it can produce frames (say you're processing in the same loop), produced frames will queue up in the driver and you'll get latency . Have this thread read from the camera and keep the most recent frame (in a variable), and do nothing else.

Whenever the mediapipe thread is ready for (more) work, it gets the current frame from the camera thread. Do that by simply accessing the variable. It can then work and send its conclusions to the game, either by setting variables or by posting to Queue objects (python comes with a queue module). When reading from a queue that may not contain an object, but you don't wanna block until there is an element added to it, you need to specify a zero timeout in the "get" call and catch the "empty" exception.