Twisted `callInThread` is blocking

Question

I have a Twisted based program that I have simplified in the following excerpt:

    class PacketSender(object):

        def __init__(self, protocol_instance):
             self._stop = False
             self._protocol = protocol_instance

        def send(self, pkti):
            try:
                pkti1 = gen_packets.next()
                reactor.callInThread(self.write_packet, pkti)
            except StopIteration:
                self._stop = True

            do_some_stuff()

            if not self._stop:
                reactor.callLater(pkti.iat, self.send, pkti1)

        def write_packet(self, pkt):
            self.protocol.transport.write(pkt)
    ...
reactor.run()

Summarizing a lot, the write_packet method will call the transport.write method of a certain protocol instance I have passed to the constructor class A . Note this is a recursive implementation where The method send calls itself to get the following packet to send over the protocol. The problem is that when it calls send delayed pkt.iat seconds, the execution of send will die at that callInThread call, which is supposed to spawn a new thread and continue with the execution of do_some_stuff , right?? All this code is actually running in another twisted thread, meaning that at some point we are calling from outside callInThread(packet_sender.send, pkt_0) .

Do you have any clues about what might be happening? Is there something I'm missing about how Twisted threads work?

Answer 1

self.protocol.transport.write is a method on an object in the reactor. You're not allowed to call this method in a non-reactor thread; the behavior is undefined. Blocking is one potential result of doing undefined things. You also aren't allowed to have more than one reactor thread in a process.

The twisted documentation on threading covers this; perhaps you should review it. You almost certainly do not need to use threads at all for what you're trying to do ( transport.write is already non-blocking so trying to do it in a thread would not help anyway); if you need threads for some reason not shown here perhaps you should ask a more specific question about how to avoid them for that task.