Python Queue.Queue 在线程 TCP 流处理程序中不起作用

Question

i'm trying to make a threaded tcp stream handler class thread and the main thread talk, however Queue.Queue is not doing what i need it too, the server receives data from another program, and i just want to pass it into the main thread for processing here is the code i have so far:

EDIT: Updated using proposed answer, queue still does nothing i cant even get any print commands to work after i does the self.queue.put(info) its like the whole thing dies. However it doesn't give me any errors, but the stream handler/thread the data comes in on dies after i put the info into the queue.

class ThreadedTCPStreamHandler(SocketServer.StreamRequestHandler):

    def __init__(self, request, client_address, server):
         SocketServer.StreamRequestHandler.__init__(self, request, client_address, server)
         self.queue = self.server.queue

    def handle(self):
                while True:
                    try:
                        self.data = self.rfile.readline().strip()
                        cur_thread = threading.current_thread()
                        command = self.data[0:2]
                        if command == "nr":
                            info = self.data[2:]
                            t1 = info.split("|")
                            title = t1[0]
                            self.queue.put(info)

class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):

        def __init__(self, server_address, RequestHandlerClass, bind_and_activate=True, queue=None):
            self.queue = queue
            SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass, bind_and_activate=bind_and_activate)

And in a main thread class i have:

q = Queue.Queue()
server = ThreadedTCPServer((HOST, PORT), ThreadedTCPStreamHandler, queue=q)
ip, port = server.server_address

server_thread = threading.Thread(target=server.serve_forever)
server_thread.daemon = True
server_thread.start()
while True:
        try:
            item = q.get()
            print item
            q.task_done()
            q.join()
        except KeyboardInterrupt:
            server.shutdown()
            sys.exit(0)

However nothing is printing out and it wont pass the data, i know i'm doing something wrong, i just dont know what it is. If anyone has any insight that could help i would greatly appreciate it, thank you.

Answer 1

Both the stream handler and the main thread need to be using the same Queue object for this to work. You're creating separate queue objects in each place. Either use queue as a global variable, if both the client and server code are in the same file, or do this:

class ThreadedTCPStreamServer(ThreadingMixin, TCPServer):
    def __init__(self, server_address, RequestHandlerClass, bind_and_activate=True,
                 queue=None):
        self.queue = queue
        TCPServer.__init__(self, server_address, RequestHandlerClass,
                           bind_and_activate=bind_and_activate)

class ThreadedTCPStreamHandler(SocketServer.StreamRequestHandler):
    def __init__(self, request, client_address, server):
        self.queue = server.queue
        StreamRequestHandler.__init__(self, request, client_address, server)

    def handle(self):
        while True:
            self.data = self.rfile.readline().strip()
            if not self.data:
                break
            cur_thread = threading.current_thread()
            command = self.data[0:2]
            if command == "nr":
                info = self.data[2:]
                t1 = info.split("|")
                title = t1[0]
                self.queue.put(info)
                self.finish()


q = Queue.Queue()
server = ThreadedTCPServer((HOST, PORT), ThreadedTCPStreamHandler, queue=q)
ip, port = server.server_address

server_thread = threading.Thread(target=server.serve_forever)
server_thread.daemon = True
server_thread.start()
while True:
        try:
            item = q.get()
            print item
            q.task_done()
            q.join()
        except KeyboardInterrupt:
            server.shutdown()
            sys.exit(0)

Answer 2

Found I had to adapt a couple of things from the accepted answer. This example works (at least in python2.7).

After starting the server you can test by repeatedly sending with netcat or similar:

echo "nrtest" | nc -4 127.0.0.1 50514

One can also send with the logger command on linux, but then the handler must look for other strings (this example checks that line starts with "nr", as pr OPs request).

logger -T -n 127.0.0.1 -P 50514 "Test message"

The result is as follows - broken into sections for readability.

Imports:

"""
Based on https://stackoverflow.com/a/25246157/2045924
"""
import threading
import Queue
import sys
import SocketServer
from SocketServer import StreamRequestHandler
from SocketServer import TCPServer
from SocketServer import ThreadingMixIn

Adding queue to TCPServer:

class ThreadedTCPStreamServer(ThreadingMixIn, TCPServer):
    """ThreadedTCPStreamServer."""
    def __init__(self, server_address, RequestHandlerClass, bind_and_activate=True,
                 queue=None):
        self.queue = queue
        TCPServer.__init__(self, server_address, RequestHandlerClass,
                           bind_and_activate=bind_and_activate)

Add queue to StreamRequestHandler, tell "handle" what to do, and let "finish" just pass.

class ThreadedTCPStreamHandler(SocketServer.StreamRequestHandler):
    """ThreadedTCPStreamHandler."""
    def __init__(self, request, client_address, server):
        self.queue = server.queue
        StreamRequestHandler.__init__(self, request, client_address, server)

    def handle(self):
        while True:
            self.data = self.rfile.readline().strip()
            if not self.data:
                break
            __cur_thread = threading.current_thread()
            self.finish()
            command = self.data[0:2]
            if command == "nr":
                info = self.data[2:]
                __t1 = info.split("|")
                __title = __t1[0]
                self.queue.put(info)
                self.finish()
    def finish(self):
        pass

Define globals and start.

HOST = '127.0.0.1'
PORT = 50514

# QUEUE must be defined _outside_.
QUEUE = Queue.Queue()
QTIMEOUT = 1

# Setup instance of my own ThreadedTCPStreamServer with my own ThreadedTCPStreamHandler.
SERVER = ThreadedTCPStreamServer((HOST, PORT), ThreadedTCPStreamHandler, queue=QUEUE)
__IP, __PORT = SERVER.server_address

# Start the server
SERVER_THREAD = threading.Thread(target=SERVER.serve_forever)
SERVER_THREAD.daemon = True
SERVER_THREAD.start()

while True:
    try:
        #ITEM = QUEUE.get(block=True, timeout=QTIMEOUT)
        ITEM = QUEUE.get()
        print 'qsize({s}): {i}'.format(s=QUEUE.qsize(), i=ITEM)
        QUEUE.task_done()
        QUEUE.join()
    except KeyboardInterrupt:
        SERVER.shutdown()
        sys.exit(0)
    #except Queue.Empty:
    #    print 'Got Queue.Empty after waiting for {t}. Continuing...'.format(t=QTIMEOUT)
    #    continue

Now start sending something with nc or a similar tool.

To end the program, press <ctrl>+C .

KeyboardInterrupt will "work" if you send a packet.

1. Press +C
2. Send an extra packet
3. Voila, program finishes