Python how to kill threads blocked on queue with signals?

Question

I start a bunch of threads working on a queue and I want to kill them when sending the SIGINT (Ctrl+C). What is the best way to handle this?

targets = Queue.Queue()
threads_num = 10
threads = []

for i in threads_num:
    t = MyThread()
    t.setDaemon(True)
    threads.append(t)
    t.start()

targets.join()

Answer 1

If you are not interested in letting the other threads shut down gracefully, simply start them in daemon mode and wrap the join of the queue in a terminator thread.

That way, you can make use of the join method of the thread -- which supports a timeout and does not block off exceptions -- instead of having to wait on the queue's join method.

In other words, do something like this:

term = Thread(target=someQueueVar.join)
term.daemon = True
term.start()
while (term.isAlive()):
    term.join(3600)

Now, Ctrl+C will terminate the MainThread whereupon the Python Interpreter hard-kills all threads marked as "daemons". Do note that this means that you have to set "Thread.daemon" for all the other threads or shut them down gracefully by catching the correct exception (KeyboardInterrupt or SystemExit) and doing whatever needs to be done for them to quit.

Do also note that you absolutely need to pass a number to term.join() , as otherwise it will, too, ignore all exceptions. You can select an arbitrarily high number, though.

Answer 2

Isn't Ctrl + C SIGINT ?

Anyway, you can install a handler for the appropriate signal, and in the handler:

• set a global flag that instructs the workers to exit, and make sure they check it periodically
• or put 10 shutdown tokens on the queue, and have the workers exit when they pop this magic token
• or set a flag which instructs the main thread to push those tokens, make sure the main thread checks that flag

etc. Mostly it depends on the structure of the application you're interrupting.

Answer 3

One way to do it is to install a signal handler for SIGTERM that directly calls os._exit(signal.SIGTERM) . However unless you specify the optional timeout argument to Queue.get the signal handler function will not run until after the get method returns. (That's completely undocumented; I discovered that on my own.) So you can specify sys.maxint as the timeout and put your Queue.get call in a retry loop for purity to get around that.

Answer 4

This is how I tackled this.

class Worker(threading.Thread):
    def __init__(self):
        self.shutdown_flag = threading.Event()
    def run(self):
        logging.info('Worker started')
        while not self.shutdown_flag.is_set():
            try:
                task = self.get_task_from_queue()
            except queue.Empty:
                continue
            self.process_task(task)

    def get_task_from_queue(self) -> Task:
        return self.task_queue.get(block=True, timeout=10)
    def shutdown(self):
        logging.info('Shutdown received')
        self.shutdown_flag.set()

Upon receiving a signal the main thread sets the shutdown event on workers. The workers wait on a blocking queue, but keep checking every 10 seconds if they have received a shutdown signal.

Answer 5

Why don't you set timeouts for any operation on the queue? Then your threads can regular check if they have to finish by checking if an Event is raised.

Answer 6

I managed to solve the problem by emptying the queue on KeyboardInterrupt and letting threads to gracefully stop themselves.

I don't know if it's the best way to handle this but is simple and quite clean.

targets = Queue.Queue()
threads_num = 10
threads = []

for i in threads_num:
    t = MyThread()
    t.setDaemon(True)
    threads.append(t)
    t.start()

while True:
    try:
        # If the queue is empty exit loop
        if self.targets.empty() is True:
            break

    # KeyboardInterrupt handler
    except KeyboardInterrupt:
        print "[X] Interrupt! Killing threads..."
        # Substitute the old queue with a new empty one and exit loop
        targets = Queue.Queue()
        break

# Join every thread on the queue normally
targets.join()