Kill a worker thread after a certain time in python2.7

Question

I'm working on a Python 2.7 script using threading.

There is one global connection object, which has to be used by each thread.

Code Example:

from threading import Thread
import time


class Connection:
    def __init__(self):
        self.connected = True

    def send_command(self, command):
        return str(command)+' result'


class Config:
    def __init__(self):
        self.conn = Connection()

    def do_remote_config(self):
        time.sleep(2)
        return self.conn.send_command('my config')

    def do_other_remote_config(self):
        time.sleep(2)
        return self.conn.send_command('my other config')


class Executor:
    def execute(self):
        config = Config()

        worker1 = Worker(config.do_remote_config)
        worker1.start()
        worker1.join()
        print(worker1.result)

        worker2 = Worker(config.do_other_remote_config)
        worker2.start()
        worker2.join()
        print(worker2.result)


class Worker(Thread):
    def __init__(self, method):
        super(Worker, self).__init__()
        self.result = None
        self.method = method

    def run(self):
        try:
            self.result = self.method()
        except Exception as ex:
            self.result = ex


if __name__ == "__main__":
    e = Executor()
    e.execute()

In order to ensure that none of the threads runs for more than 10 minutes, I wanted to kill each thread in case the time limit is reached. Unfortunately, it turns out that Python threads cannot be killed.

Thread Kill Pill Option:

Due to the actual complexity of the worker threads, it is unfortunately not possible to build some kind of kill-trigger, which lets the worker thread end himself. So, it seems that I really need to get rid of threading here because threads by nature cannot be killed.

Multiprocess Option:

Using the multiprocess module, different processes could be used. Those could then be killed after a certain time. However I did not find a way to pass on my connection object in such a way that it can be used by several processes.

Remote Procedure Calls (RPC) option:

RPCs seem to introduce an unnecessary level of complexity and the kill switch could presumably still not be implemented.

Question:

Which Python technologies would work best in order to being able to use the connection object with all workers while ensuring that each worker can reliably be killed after 10 minutes?

Thanks very much!

Answer 1

Too long for a comment, too abstract for an answer.

I would say that multiprocessing is the way to go if you wish to be able to interrupt processing in a random moment without revamping the whole processing. All other methods demand some sort of cooperation from threads being interrupted.

Certainly splitting the whole process into pieces demand some processing changes as well. All shared file-like resources (opened files, sockets, pipes) are to be opened before the forking of processes and carefully orchestrated. Probably the safest approach would be like this:

you have a master socket being listen() ed by a master process. The master also runs a workers pool. It's essential to create the master socket before the pool, to make the socket available to the workers.

The master delivers new job tasks to the workers and receives results if needed via multiprocessing primitives .

When a new client arrives, the master orders a selected worker from the pool to accept() the connection and returns to back to waiting for new clients and other master's activities. The worker accept() s the connection thus creating a private socket to communicate with a particular client, no other workers can and should access the client socket.

If workers need to communicate with each other, all necessary communication primitives must be created before the pool and distributed among the workers by the master.