具有许多线程的Python生产者/消费者

Question

I wrote this small application to solve the classical Producer/Consumer problem in Python. I know that I could use a thread safe Queue mechanism to solve this problem, but I was interested in solving this problem on my own for learning.

from threading import Thread, Condition
from collections import deque
import random
import time

tasks = deque()
condition = Condition()

class Consumer(Thread):

    def process_task(self, task):
        self.log("Completed task " + str(task) )

    def get_task(self):
        # this is the only method that access the global object
        global tasks
        condition.acquire()
        if len(tasks) is 0:
            self.log("Nothing in the Queue... Sleeping!")
            condition.wait()
            return None
        task = tasks.popleft()
        condition.release();
        return task

    def sleep(self):
        self.log("Sleeping")
        time.sleep(1)

    def execute(self):
        while True:
            task = self.get_task()
            if task:
                self.process_task(task)

    def run(self):
        self.log("Started")
        self.execute()

    def log(self, msg):
        print("[ Consumer: {0} ] Consumed {1}".format(self.getName(), msg))

class Producer(Thread):

    def create_tasks(self):
        return [random.randint(0,100) for x in range(100)]

    def add_tasks(self, new_tasks):
        global tasks
        condition.acquire()
        tasks.extend(new_tasks)
        self.log("Produced " + str(len(new_tasks)) + "elements")
        self.log("Queue length : " + str(len(tasks)))
        self.notify_all()
        condition.release();

    def notify_all(self):
        condition.notify_all()
        self.log("Aweking consumers")

    def sleep(self):
        self.log("Sleeping")
        time.sleep(1)

    def execute(self):
        while True:
            new_tasks = self.create_tasks()
            self.add_tasks(new_tasks)
            self.sleep()

    def run(self):
        self.log("Started")
        self.execute()

    def log(self, msg):
        print("[ Producer: {0} ] {1}".format(self.getName(), msg))


Producer().start()
Producer().start()

Consumer().start()
Consumer().start()
Consumer().start()

The application appears to work properly. What I mean by this is that I haven't experienced any deadlock or weird exceptions. However, the result I've obtained is not what I was expecting as the work is not spread among the different consumers at all. I was expecting that all three consumer thread would have done some work in parallel, but this didn't happen. This is an extract from the logs :

[ Consumer: Thread-4 ] Consumed Completed task 42
 ... 98 times the above line with a different task ( always thread 4)  ...
[ Consumer: Thread-4 ] Consumed Completed task 22
[ Consumer: Thread-4 ] Consumed Nothing in the Queue... Sleeping!
[ Consumer: Thread-5 ] Consumed Nothing in the Queue... Sleeping!
[ Consumer: Thread-3 ] Consumed Nothing in the Queue... Sleeping!
[ Producer: Thread-1 ] Produced 100elements
[ Producer: Thread-1 ] Queue length : 100
[ Producer: Thread-1 ] Aweking consumers
[ Producer: Thread-1 ] Sleeping
[ Consumer: Thread-3 ] Consumed Completed task 87
 ... 98 times the above line with a different task ( always thread 3)...
[ Consumer: Thread-3 ] Consumed Completed task 20
[ Consumer: Thread-3 ] Consumed Nothing in the Queue... Sleeping!
[ Consumer: Thread-4 ] Consumed Nothing in the Queue... Sleeping!
[ Consumer: Thread-5 ] Consumed Nothing in the Queue... Sleeping!

As you can see from the above logs, all work is carried out by one single thread whereas the other 2 thread don't do anything. It looks like they never had the chance to run. I tried to put asleep a consumer thread after it has executed a task, but the behaviour didn't change.

Is there a bug in my code that I don't see? Could be this behaviour related to the famous "GIL"?

Thank you for your help.

Answer 1

What you're seeing isn't really a coding problem, its just that the same thread is usually winning the race to re-acquire the condition mutex. There's enough time for the thread to run process_task and still win the race, because process_task is hardly doing anything. I don't think this is actually related to the GIL; I ran the code in Jython, which has no GIL, and the same behavior occurred.

There's definitely no bug, though. When I ran your code in CPython, I occasionally had more than one thread end up consuming:

[ Producer: Thread-1 ] Started
[ Producer: Thread-1 ] Produced 100elements
[ Producer: Thread-1 ] Queue length : 100
[ Producer: Thread-1 ] Aweking consumers
[ Producer: Thread-1 ] Sleeping
[ Producer: Thread-2 ] Started
[ Consumer: Thread-3 ] Consumed Started
[ Producer: Thread-2 ] Produced 100elements
[ Producer: Thread-2 ] Queue length : 200
[ Consumer: Thread-5 ] Consumed Started
[ Producer: Thread-2 ] Aweking consumers
[ Consumer: Thread-4 ] Consumed Started
[ Producer: Thread-2 ] Sleeping
[ Consumer: Thread-5 ] Consumed Completed task 53
[ Consumer: Thread-4 ] Consumed Completed task 73
[ Consumer: Thread-5 ] Consumed Completed task 83
[ Consumer: Thread-4 ] Consumed Completed task 71
[ Consumer: Thread-5 ] Consumed Completed task 67
[ Consumer: Thread-4 ] Consumed Completed task 7
[ Consumer: Thread-5 ] Consumed Completed task 34
[ Consumer: Thread-4 ] Consumed Completed task 68
[ Consumer: Thread-5 ] Consumed Completed task 15
[ Consumer: Thread-4 ] Consumed Completed task 29
[ Consumer: Thread-5 ] Consumed Completed task 20
... (4 and 5 continue to switch off)

Also, I get very normal results if I call self.sleep() inside of process_task , which simulates some real work actually happening, and therefore makes for a much more realistic test:

[ Producer: Thread-1 ] Started
[ Producer: Thread-2 ] Started
[ Producer: Thread-1 ] Produced 100elements
[ Producer: Thread-1 ] Queue length : 100
[ Producer: Thread-1 ] Aweking consumers
[ Consumer: Thread-5 ] Consumed Started
[ Consumer: Thread-3 ] Consumed Started
[ Producer: Thread-1 ] Sleeping
[ Producer: Thread-2 ] Produced 100elements
[ Consumer: Thread-4 ] Consumed Started
[ Producer: Thread-2 ] Queue length : 200
[ Producer: Thread-2 ] Aweking consumers
[ Producer: Thread-2 ] Sleeping
[ Consumer: Thread-3 ] Consumed Sleeping
[ Consumer: Thread-5 ] Consumed Sleeping
[ Consumer: Thread-4 ] Consumed Sleeping
[ Producer: Thread-1 ] Produced 100elements
[ Consumer: Thread-3 ] Consumed Completed task 85
[ Consumer: Thread-5 ] Consumed Completed task 31
[ Producer: Thread-1 ] Queue length : 297
[ Consumer: Thread-4 ] Consumed Completed task 62
[ Producer: Thread-1 ] Aweking consumers
[ Producer: Thread-1 ] Sleeping
[ Producer: Thread-2 ] Produced 100elements
[ Producer: Thread-2 ] Queue length : 397
[ Producer: Thread-2 ] Aweking consumers
[ Producer: Thread-2 ] Sleeping
[ Consumer: Thread-3 ] Consumed Sleeping
[ Consumer: Thread-5 ] Consumed Sleeping
[ Consumer: Thread-4 ] Consumed Sleeping
[ Producer: Thread-1 ] Produced 100elements
[ Producer: Thread-1 ] Queue length : 494
[ Consumer: Thread-3 ] Consumed Completed task 99
[ Consumer: Thread-4 ] Consumed Completed task 58
[ Producer: Thread-1 ] Aweking consumers
[ Consumer: Thread-5 ] Consumed Completed task 18