Python Queue - Why Is It Getting Still Growing?

Question

I have a server taking TCP data and spawning threads to parse up and deal with the data. I noticed if it starts to get too many connections something goes wrong and it starts grinding to a halt so after hunting around I discovered that my queue steadily gets more and more data even though I'm constantly using a q.get().

Snippet of the Queue emptying thread:

def QDUmp():
    while 1:
        if not q.empty():

            data= str(q.get())
            ParseThread = Thread(target = ParseFunction(data))
            ParseThread.start()

CheckQ = Thread(target = QDUmp)
CheckQ.start()

The data is constantly being added by the TCP just with a standart put like this pseudo:

q = Queue.Queue()

For eachdata from Socket:
  q.put(eachdata)

So... data comes in and data goes out but it's not going out quick enough.

The longer it runs the more and more data is sitting in that q. As I understood it q.get() would of removed the data from there and for every bit of data there it should be spawning a thread to remove it.

Can anyone shed some light on how I'm winding up with more and more data in that queue?

Is it just coming in faster than while 1: is able to loop around and check it?

Answer 1

I believe the TCP data receiver is too fast than your data processor. You can add a counter to count how many data in and how many data out, so you can know whether the queue is growing continuously.