How to ensure only one thread runs at a time, the most recent data?

Question

I have a Flask app with a certain URL route that, when called, invokes a function inside a process via the threading library. This function contains a while True loop that, inside, has a DB query.

That same function may be invoked again, but the DB may change, which now results in a different thread. I don't want this however, as it causes a thread to have old data.

I want one thread.

Here's my current approach:

@mod.route('/api/start-zmq-listener')
def startZMQListener():
    try:
        thread = threading.Thread(target=enableZMQ, name="enableZMQ", kwargs={
            'username': 'test',
            'stopFlag': stopFlag
        })
        thread.start()
        return success_response('ok')

    except Exception as e:
        return error_response(e)


def enableZMQ(username, stopFlag):
    if not stopFlag:
        context = zmq.Context()

        listen_on = 'tcp://' + ENGINE_IP 

        sock = context.socket(zmq.SUB)
        sock.setsockopt(zmq.SUBSCRIBE, b"")
        sock.connect(listen_on)
        print ('listening on ' + listen_on)

        while True:
            print MONGODB QUERY

Let's say the "MONGODB QUERY" is the same each time, but the data it queries is different.

How do I make sure there is only one process running at all times that uses the most recent MongoDB query?

Answer 1

One possible solution is have the data always ready in ram.

You could have a callback in your flask app that fills a global value with data. This callback would be called from another thread that runs on a interval.

Something like this:

from threading import Thread, Lock
# Add flask stuff

global value_store = [] # Some data
global mutex = Lock()
INTERVAL = 1  # second
global should_query = False

@mod.route('/api/getSomeData')
def get_some_data():
    global mutex
    with mutex
        global value_store
        return jsonify(value_store)

def callback(data):
    global mutex 
    with mutex:
        global value_store
        value_store = data

def query_repeat():
    global should_query
    while should_query:
        # query db here
        raw data = query_database()

        # filter/process data
        data = filter_data(raw_data)

        # call callback
        callback(data)

        # sleep and repeat
        time.sleep(INTERVAL)


t = Thread(target = query_repeat, args = (,))
should_query = True
t.start()

I may have some scope errors with function calls, but thats the general idea.