Gevent infinite queue consumer?

Question

I am trying to have a consumer on a separate thread that never dies. The consumer can get arbitrary number of tasks to execute.

I have been fiddling with asyncio and gevent but haven't managed to get anything that works. Below is a very simplified example of what I'm trying to do.

q = gevent.queue.SimpleQueue()


def run_task(task):
    print(f"Started task {task} @ {time.time()}")
    gevent.sleep(1)
    print(f"Finished task {task} @ {time.time()}")


def consumer():
    while True:
        task = q.get()
        print(f"Dequed task {task} for consumption")
        gevent.spawn(run_task, task)


q.put(1)
q.put(2)
q.put(3)
consumer()

Output

Dequed task 1 for consumption
Dequed task 2 for consumption
Dequed task 3 for consumption

Clearly run_task is never executed for the tasks. I can use join() but then the tasks are run in sequence. Using joinall() doesn't seem to be a viable solution either since the queue will keep getting arbitrary tasks.

Any ideas how to approach this?

Answer 1

There are two issues with your code:

1. SimpleQueue is an alias of the Python standard library queue which requires gevent monkey patching to use.

2. You are running your consumer on the main thread.

This code works as expected:

import gevent
import gevent.queue
import time

q = gevent.queue.Queue()

def run_task(task):
    print(f"Started task {task} @ {time.time()}")
    gevent.sleep(1)
    print(f"Finished task {task} @ {time.time()}")

def consumer():
    while True:
        task = q.get()
        print(f"Dequed task {task} for consumption")
        gevent.spawn(run_task, task)

q.put(1)
q.put(2)
q.put(3)

gevent.spawn(consumer)

gevent.sleep(1000)