How to implement threaded socket.recv() in python?

Question

I have a number of devices from which i need to get status updates. A socket object is all I have, and socket.recv() is all I need to get the status. Put into a single threaded application, no problems occur:

class Device:
    def receive(self):
        log.debug("receive waiting: %r", self.device_id)
        try:
            packet = self.socket.recv(255)
        except Exception as e:
            self.report_socket_error(e)
            self.reconnect()
        log.debug("received response: %r", self.device_id)
d = Device()
d.connect()
while True:
    d.receive()

However, the same code wrapped in a threading.Thread class causes deadlocks and funny behaviour. Wrapping it with locks didn't change anything. I traced the problem down to be the socket.recv() call...So, how to implement multiple threads where each thread owns one socket (1 thread owns exclusively 1 socket), which are able to wait for data simultaneously?

Thanks in advance

Answer 1

How many different sockets do you have to read from?

• If the answer is "just one", then use just one thread. Adding another helps you in no way and only complicates your life, as you found out.

• If the answer is "several", than one way to organize this is indeed to have a thread per socket. recv is a blocking operation, which makes a thread an attractive option to organize code. Each thread owns a separate socket and reads from it at its leisure. You should have no problems and deadlocks with this.

  Locks are unnecessary as long as no resources are shared. Even if you do share resources (logging, some data store, etc.) don't just use simple locks - Python has higher-level utilities for that like the Queue module.

Answer 2

I know this does not answer your question on how to fix your deadlock problem, however it appears as your use of threads is overhead in your case:

You can just use one thread in which you use select() to find out which socket has available data and then handle the reported data. Unless the handling takes long or your protocol is more complicated select should be just fine and avoid all threading issues.

Have a look at http://docs.python.org/howto/sockets.html#non-blocking-sockets for more details.