Reliable queue based inter-thread messaging in Python 3

Question

I'm trying to use the queue.Queue() class to communicate between threads. I never had a problem with it in 2.x, but lately, it has seemed really unreliable to me in 3.x.

I start with a queue, as a global variable in a module. Both threads are in this module.

q = queue.Queue()

Thread 1 has a put command:

global q
q.put(“Hello”)

Thread 2 has a get command:

global q
z = q.qsize()
q.get_nowait()

I can see in the pycdev debugger that 1 is not empty after the put, but the get is still returning empty and z is 0 before the get command. This seems really strange to me. I suspected mismatched scopes, but can't figure out where this might be happening. Does anyone have an idea conditions could cause this?

Or is there a more reliable, recommended way of doing this?

Answer 1

Update. I found the source of the problem and it is reproducable. Here is runnable, sample code:

from bottle import route, run, response
import time
import sys
import threading
import queue

engineStartQueue = queue.Queue()

class EngineStarter(threading.Thread):

    def run(self):
        try:
            global engineStartQueue
            startParams = engineStartQueue.get_nowait()

            #let url handler return 
            engineStartQueue.put_nowait([200, "Starting..."])
            time.sleep(10.0)
        except Exception as e:
            fullerror = sys.exc_info()
            errorID = str(fullerror[0])
            errorMsg = str(fullerror[1])
            alertMessage = "%s, %s" %(errorID, errorMsg)
            engineStartQueue.push([500, alertMessage])


@route('/admin/start', method='POST')
def start():   
    try:
        global engineStartQueue
        engineStartQueue.put("Hello")
        starter = EngineStarter()
        starter.start()

        time.sleep(3.0)  #Give the EngineStarter instance a couple of seconds to read the queue and respond
        returnParams = engineStartQueue.get_nowait()

        response.status = returnParams[0]
        return(returnParams[1])
    except Exception as e:
        fullerror = sys.exc_info()
        errorID = str(fullerror[0])
        errorMsg = str(fullerror[1])
        returnStr = "Failed to start:  %s, %s" %(errorID, errorMsg)
        response.status = 500
        return returnStr

if __name__ == '__main__':
    run(host='localhost', port=8080)

It is a bottle app, with a fire and forget REST call. There is url handler, a threadable object and a globally visible queue.

The URL handler 1 - creates an instance of the threadable object, EngineStarter 2 - Passes "Hello" to the queue 3 - Starts the thread 4 - Waits 3 seconds and gets the response from the EngineStarter thread 5 - Returns the http response code to the caller

The EngineStarter thread: 1 - Reads the queue 2 - Responds with an http ok if the thread is not empty and a 500 error otherwise 3 - The engine starter thread now waits 10 seconds, to simulate a long running background task on the server

If I start the script in the pydev debugger and then call the handler without any breakpoints set, I get an http 200 response after 3 seconds. This means that the scope was shared.

If I set a breakpoint before looking into the queue, it is empty. If I set the breakpoint after looking into the queue, then get_nowait() picks up "Hello".

It seems as if breakpoints in the pydev debugger are affecting the scope somehow.