Implement a push/pull (or queue) with redis in Python

Question

I'm currently working on a project where I have a lot of tasks that can be parallelized. I'm using redis to store data, and I'd also like to use to handle my jobs queue to.

At first, I've used the PubSub pattern described in Redis doc. The issue here is that it sends every message to every worker where I'd like one message per worker. I have, for the moment, circumvented the problem by adding status (in a DB) for each of the messages but that's clearly not optimal.

Is there a way to easily replace this pubsub pattern with a suitable pattern ?

Answer 1

Not sure it that's the most efficient way of implementing a queue with redis but it's definitely a simple way of doing it!

import time
from redis import Redis

REDIS_CLIENT = Redis()

def push(*values):
    REDIS_CLIENT.rpush('QUEUE', *values)

def pull():
    while True:
        msg = REDIS_CLIENT.rpop('QUEUE')
        if msg is None:
            time.sleep(0.1)
            continue

        work(msg)