Python-单独线程上的非空共享列表显示为空

Question

I've two classes - MessageProducer and MessageConsumer.

MessageConsumer does the following:

1. receives messages and puts them in its message list "_unprocessed_msgs"
2. on a separate worker thread, moves the messages to internal list "_in_process_msgs"
3. on the worker thread, processes messages from "_in_process_msgs"

On my development environment, I'm facing issue with #2 above - after adding a message by performing step#1, when worker thread checks length of "_unprocessed_msgs", it gets it as zero. When step #1 is repeated, the list properly shows 2 items on the thread on which the item was added. But in step #2, on worker thread, again the len(_unprocessed_msgs) returns zero.

Not sure why this is happening. Would really appreciate help any help on this.

I'm using Ubuntu 16.04 having Python 2.7.12.

Below is the sample source code. Please let me know if more information is required.

import threading
import time
class MessageConsumerThread(threading.Thread):
    def __init__(self):
        super(MessageConsumerThread, self).__init__()
        self._unprocessed_msg_q = []
        self._in_process_msg_q = []
        self._lock = threading.Lock()
        self._stop_processing = False

    def start_msg_processing_thread(self):
        self._stop_processing = False
        self.start()

    def stop_msg_processing_thread(self):
        self._stop_processing = True

    def receive_msg(self, msg):
        with self._lock:
            LOG.info("Before: MessageConsumerThread::receive_msg: "
                     "len(self._unprocessed_msg_q)=%s" %
                     len(self._unprocessed_msg_q))
            self._unprocessed_msg_q.append(msg)
            LOG.info("After: MessageConsumerThread::receive_msg: "
                     "len(self._unprocessed_msg_q)=%s" %
                     len(self._unprocessed_msg_q))

    def _queue_unprocessed_msgs(self):
        with self._lock:
            LOG.info("MessageConsumerThread::_queue_unprocessed_msgs: "
                     "len(self._unprocessed_msg_q)=%s" %
                     len(self._unprocessed_msg_q))
            if self._unprocessed_msg_q:
                LOG.info("Moving messages from unprocessed to in_process queue")
                self._in_process_msg_q += self._unprocessed_msg_q
                self._unprocessed_msg_q = []
                LOG.info("Moved messages from unprocessed to in_process queue")

    def run(self):
        while not self._stop_processing:
            # Allow other threads to add messages to message queue
            time.sleep(1)

            # Move unprocessed listeners to in-process listener queue
            self._queue_unprocessed_msgs()

            # If nothing to process continue the loop
            if not self._in_process_msg_q:
                continue

            for msg in self._in_process_msg_q:
                self.consume_message(msg)

            # Clean up processed messages
            del self._in_process_msg_q[:]

    def consume_message(self, msg):
        print(msg)


class MessageProducerThread(threading.Thread):
    def __init__(self, producer_id, msg_receiver):
        super(MessageProducerThread, self).__init__()
        self._producer_id = producer_id
        self._msg_receiver = msg_receiver

    def start_producing_msgs(self):
        self.start()

    def run(self):
        for i in range(1,10):
            msg = "From: %s; Message:%s" %(self._producer_id, i)
            self._msg_receiver.receive_msg(msg)


def main():
    msg_receiver_thread = MessageConsumerThread()
    msg_receiver_thread.start_msg_processing_thread()

    msg_producer_thread = MessageProducerThread(producer_id='Producer-01',
                                                msg_receiver=msg_receiver_thread)
    msg_producer_thread.start_producing_msgs()
    msg_producer_thread.join()
    msg_receiver_thread.stop_msg_processing_thread()
    msg_receiver_thread.join()

if __name__ == '__main__':
    main()

Following is the log the I get:

INFO: MessageConsumerThread::_queue_unprocessed_msgs: len(self._unprocessed_msg_q)=0
INFO: Before: MessageConsumerThread::receive_msg: len(self._unprocessed_msg_q)=0
INFO: After: MessageConsumerThread::receive_msg: **len(self._unprocessed_msg_q)=1**
INFO: MessageConsumerThread::_queue_unprocessed_msgs: **len(self._unprocessed_msg_q)=0**
INFO: MessageConsumerThread::_queue_unprocessed_msgs: len(self._unprocessed_msg_q)=0
INFO: Before: MessageConsumerThread::receive_msg: len(self._unprocessed_msg_q)=1
INFO: After: MessageConsumerThread::receive_msg: **len(self._unprocessed_msg_q)=2**
INFO: MessageConsumerThread::_queue_unprocessed_msgs: **len(self._unprocessed_msg_q)=0**

Answer 1

This is not a good desing for you application. I spent some time trying to debug this - but threading code is naturally complicated, so we should try to descomplicate it, instead of getting it even more confure.

When I see threading code in Python, I usually see it written a in a procedural form: a normal function that is passed to threading.Thread as the target argument that drives each thread. That way, you don't need to write code for a new class that will have a single instance.

Another thing is that, although Python's global interpreter lock itself guarantees lists won't get corrupted if modified in two separate threads, lists are not a recomended "thread data passing" data structure. You probably should look at threading.Queue to do that

The thing is wrong in this code at first sight is probably not the cause of your problem due to your use of locks, but it might be. Instead of

self._unprocessed_msg_q = []

which will create a new list object, the other thread have momentarily no reference too (so it might write data to the old list), you should do:

self._unprocessed_msg_q[:]  = []

Or just the del slice thing you do on the other method.

But to be on the safer side, and having mode maintanable and less surprising code, you really should change to a procedural approach there, assuming Python threading. Assume "Thread" is the "final" object that can do its thing, and then use Queues around:

# coding: utf-8
from __future__ import print_function
from __future__ import unicode_literals

from threading import Thread
try:
    from queue import Queue, Empty
except ImportError:
    from Queue import Queue, Empty
import time
import random


TERMINATE_SENTINEL = object()
NO_DATA_SENTINEL = object()


class Receiver(object):

    def __init__(self, queue):
        self.queue = queue
        self.in_process = []

    def receive_data(self, data):
        self.in_process.append(data)

    def consume_data(self):
        print("received data:",  self.in_process)
        del self.in_process[:]

    def receiver_loop(self):
        queue = self.queue
        while True:
            try:
                data = queue.get(block=False)
            except Empty:
                print("got no data from queue")
                data = NO_DATA_SENTINEL

            if data is TERMINATE_SENTINEL:
                print("Got sentinel: exiting receiver loop")
                break

            self.receive_data(data)

            time.sleep(random.uniform(0, 0.3))
            if queue.empty():
                # Only process data if we have nothing to receive right now:
                self.consume_data()
                print("sleeping receiver")
                time.sleep(1)
        if self.in_process:
            self.consume_data()


def producer_loop(queue):
    for i in range(10):
        time.sleep(random.uniform(0.05, 0.4))
        print("putting {0} in queue".format(i))
        queue.put(i)


def main():
    msg_queue = Queue()
    msg_receiver_thread = Thread(target=Receiver(msg_queue).receiver_loop)
    time.sleep(0.1)
    msg_producer_thread = Thread(target=producer_loop, args=(msg_queue,))

    msg_receiver_thread.start()
    msg_producer_thread.start()
    msg_producer_thread.join()
    msg_queue.put(TERMINATE_SENTINEL)
    msg_receiver_thread.join()

if __name__ == '__main__':
    main()

note that since you want multiple methods in the recever thread to do things with data, I used a class - but it does not inherit from Thread, and does not have to worry about its workings. All its methods are called within the same thread: no need of locks, no worries about race conditions within the receiver class itself. For communicating outside the class, the Queue class is structured to handle any race conditions for us.

The producer loop, as it is just a dummy producer, has no need at all to be written in class form. But it would look just the same, if it had more methods.

(The random sleeps help visualize what would happen in "real world" message receiving) Also, you might want to take a look at something like: https://www.thoughtworks.com/insights/blog/composition-vs-inheritance-how-choose

Answer 2

Finally I was able to solve the issue. In the actual code, I've a Manager class that is responsible for instantiating MessageConsumerThread as its last thing in the initializer:

class Manager(object):
    def __init__(self):
        ...
        ...
        self._consumer = MessageConsumerThread(self)
        self._consumer.start_msg_processing_thread()

The problem seems to be with passing 'self' in MessageConsumerThread initializer when Manager is still executing its initializer (eventhough those are last two steps). The moment I moved the creation of consumer out of initializer, consumer thread was able to see the elements in "_unprocessed_msg_q".

Please note that the issue is still not reproducible with the above sample code. It is manifesting itself in the production environment only. Without the above fix, I tried queue and dictionary as well but observed the same issue. After the fix, tried with queue and list and was able to successfully execute the code.

I really appreciate and thank @jsbueno and @ivan_pozdeev for their time and help! Community @stackoverflow is very helpful!