如何从flask-mqtt回调中获取数据到连续的flask文本/事件流响应中？

Question

I would like to adapt this guide for real-time charts in Flask so that data received via MQTT in the backend can be passed to the frontend via Server Sent Events. For receiving MQTT data I use flask-mqtt. I can receive data from a topic with the following callback function:

@mqtt.on_message()
def handle_messages(client, userdata, message):
    print('Received message on topic {}: {}'
          .format(message.topic, message.payload.decode()))

The guide describes that data from Flask can be retrieved from the frontend using the following function:

@application.route('/chart-data')
def chart_data():
    def generate_random_data():
        while True:
            json_data = json.dumps(
                {'time': datetime.now().strftime('%Y-%m-%d %H:%M:%S'), 'value': random.random() * 100})
            yield f"data:{json_data}\n\n"
            time.sleep(1)

    return Response(generate_random_data(), mimetype='text/event-stream')

The question now is how can I transfer the data from my MQTT callback function handle_messages() into the chart_data() method so that MQTT data is transferred from the backend to the frontend via SSE?

I would like to implement the solution mostly in Flask without having to install additional servers like with Socket.io or REDIS. I also prefer not to connect to the MQTT broker in the frontend via websockets.

Answer 1

Here is an example using flask_socketio :

# app.py
import eventlet
from flask import Flask, render_template
from flask_mqtt import Mqtt
from flask_socketio import SocketIO

eventlet.monkey_patch()

app = Flask(__name__)
app.config['SECRET'] = 'my secret key'
app.config['TEMPLATES_AUTO_RELOAD'] = True
app.config['MQTT_BROKER_URL'] = 'broker.hivemq.com'
app.config['MQTT_BROKER_PORT'] = 1883
app.config['MQTT_USERNAME'] = ''
app.config['MQTT_PASSWORD'] = ''
app.config['MQTT_KEEPALIVE'] = 5
app.config['MQTT_TLS_ENABLED'] = False


mqtt = Mqtt(app)
socketio = SocketIO(app)


@app.route('/')
def index():
    return render_template('index.html')


@socketio.on('publish')
def handle_publish(data):
    mqtt.publish('top1', data['message'])


@mqtt.on_message()
def handle_mqtt_message(client, userdata, message):
    data = dict(
        topic=message.topic,
        payload=message.payload.decode()
    )
    socketio.emit('mqtt_message', data=data)


if __name__ == '__main__':
    mqtt.subscribe('top1')
    socketio.run(app, host='0.0.0.0', port=5000, use_reloader=False, debug=True)

templates/index.html :

<script type="text/javascript" src="https://code.jquery.com/jquery-3.3.1.min.js"></script>
<script type="text/javascript" src="https://cdn.socket.io/3.1.3/socket.io.min.js"></script>
<script type="text/javascript" charSet="utf-8">
    $(document).ready(function () {
        const socket = io.connect('http://' + document.domain + ':' + location.port);
        socket.on('mqtt_message', (data) => {
            const text = '(' + data['topic'] + ' qos: ' + data['qos'] + ') ' + data['payload'];
            // update chart here...
            $('#subscribe_messages').append(text + '<br/><br/>');
        })
    });
</script>
<div id="subscribe_messages"></div>

run server, open a few tabs and open browser console:

// connect and send 2 messages into top1
const s = io.connect('http://' + document.domain + ':' + location.port);
s.emit("publish", {"message": "hello1"});
s.emit("publish", {"message": "hello2"});
// you will see the messages into all tabs

About SSE without flask_socketio . You can try to use Queue for messages( is just an example ):

# app.py
import datetime
import json
import time
from queue import Queue

import eventlet
from flask import Flask, render_template, Response
from flask_mqtt import Mqtt

eventlet.monkey_patch()
QUEUE = Queue()


app = Flask(__name__)
app.config['SECRET'] = 'my secret key'
app.config['TEMPLATES_AUTO_RELOAD'] = True
app.config['MQTT_BROKER_URL'] = 'broker.hivemq.com'
app.config['MQTT_BROKER_PORT'] = 1883
app.config['MQTT_USERNAME'] = ''
app.config['MQTT_PASSWORD'] = ''
app.config['MQTT_KEEPALIVE'] = 5
app.config['MQTT_TLS_ENABLED'] = False

mqtt = Mqtt(app)
mqtt.subscribe('top1')


@app.route('/')
def index():
    return render_template('index.html')


@mqtt.on_message()
def handle_mqtt_message(client, userdata, message):
    data = dict(
        topic=message.topic,
        payload=message.payload.decode()
    )
    QUEUE.put(data)


@app.route('/publish')
def publish():
    mqtt.publish('top1', json.dumps({'dt': str(datetime.datetime.now())}).encode())
    return 'ok'


@app.route('/chart-data')
def chart_data():
    def events():
        while True:
            message = QUEUE.get()
            if message is None:
                time.sleep(1)
                continue

            yield "data: %s\n\n" % str(message)
    return Response(events(), content_type='text/event-stream')

templates/index.html :

<script type="text/javascript" src="https://code.jquery.com/jquery-3.3.1.min.js"></script>
<script type="text/javascript" charset="utf-8">
  $(document).ready(function() {
      const source = new EventSource('/chart-data');
      source.addEventListener('message', function(e) {
          // update chart here...
          $('#subscribe_messages').append(e.data + '<br/><br/>');
        }, false);
  });
</script>
<div id="subscribe_messages"></div>

Open / , open a few times in a new tab /chart-data and check / route. You'll see event-messages.

Also you can look at Quart , flask[async] , aio-pika , aioamqp .