在Python中限制线程

Question

I'm working on a project where I have a database that emits information to my Python application. Whenever my Python script receives such an 'event' it has to process the data, but also be ready to receive new events.

When operational the script will receive lots of events in a matter of milliseconds so serial processing is not an option.

Below some pseudo code to illustrate my current setup:

class Source(Thread):
# receives events and dispatches to processing threads

run():
    while True:
        data = database.receive
        for thing in data:
            DataProcessing.process(thing)

class DataProcessing():

    @Multitasking.threaded
    @staticmethod
    process(foo):
      do_something

class Multitasking():
def threaded(fn):
    def wrapper(*args, **kwargs):
        thread = Thread(target=fn, args=args, kwargs=kwargs)
        thread.setDaemon(True)
        thread.setName('worker')
        thread.start()
        return thread
    return wrapper

So here I have a Source class that acts as a listener for database events. Whenever there is an events it processes the event using the DataProcessing.process() method. I wrote a decorator/wrapper to make it a threaded so Source can go back to listening.

Now here's my problem: I use pycharm, and discovered the concurrency diagram. But when I run it something strange seems to happen.

Here the worker is the processing() method mentioned above. As you can see the amount of active threads getting larger and larger for every event received, while I'm certain the size of the data array is not getting larger.

My question: How does this diagram work? It looks like threads are re-initiated everytime a event is received, but are they? I only call thread.start() for new events.

Thanks!

Answer 1

Not sure how this specific diagram works. But I see that you start the threads, but do not join them. Until joined, the threads will be treated and marked as unfinished (though not executing). You should do thr.join() somewhere to actually finish them and to prevent the resource leak. For this, you have to keep track of them, which makes to code more complex.

Try using the undocumented multiprocessing.pool.ThreadPool , which has the same signature as from multiprocessing.Pool , but works with the threads. It can run a limited pool of the worker threads, and execute the tasks in them (or put the m to the queue). See: https://docs.python.org/2/library/multiprocessing.html