Plotly Dash/ Apache Kafka - I would like to see a graph plotting live rather than appearing when finished plotting

Question

I have viewed live updating of streaming graphs in plotly however this is not exactly what I am looking to do. I have a graph component which gets populated on the click of a button component. In the callback to update the graph an object is created and a method is called on from that class which returns a stream of 15 values using apache kafka streaming producer. I then have the apache kafka consumer in the callback which receives the values one by one. However I am trying to get these values to appear on my dash graph one by one also. I tried using an interval component but this appears to be just recalling the function every second meaning the script gets stuck recalling the function and does not do anything.

Would be grateful for any help on this and can provide more detail if necessary.

Answer 1

The consumer loop is infinite, and you cannot iterate the same consumer object twice for the same set of records. You cannot use list-comprehension, so you need to store the lists separately

The following is untested, but shows the general idea of what you would need to do

x = []
y = []

LIMIT = 15

for r in consumer_obj:

  if len(x) >= LIMIT:
    del x[0]
  if len(y) >= LIMIT:
    del y[0]

  x.append(r.value['day'])
  y.append(r.value['biomass'])
  
  # TODO: Update graph 
  data = {"data": [
          {"x": x,"y": y, 
          "type": "lines",
        }]

There is a project here using Bokeh intstead of Plotly - https://github.com/Aakash282/kafka-bokeh-dashboard

---

Or you could just use Kafka Connect to write to a SQLite database, and make your plot use that and have that update periodically

Answer 2

With Streams you will want to place the consumer in its own thread. you are already placing the producer in a thread , I would also suggest not joining the producer to the main thread either

what you want to do is place the consumer in a thread and use a queue or a thread safe data structure or locking , a queue is thread safe so you can extract the data as it comes if you like

you also want to run the threads in a def or startup , and not on a callback, each time on the callback you will just ask for the latest item on the queue and can set an interval to 100ms or so dep on pref

I have a simple example below :

• the while loop will be your callback interval - im using a while for time

• you must set the consumer outside the callback - otherwise you build it again and again, place the consumer outside the scope of the callback and run it on startup , below I used main.

    from time import sleep
    from json import dumps
    from kafka import KafkaConsumer
    from multiprocessing import Process, Queue
    import os
    
    
    
    def consumer(q):
    
        consumer_obj = KafkaConsumer('test')
        for message in  consumer_obj:
         q.put(message.value)
    
    
    if __name__ == '__main__':
        q = Queue()
        p = Process(target=consumer, args=(q,))
        p.start()
        while True:
            print(q.get())
            sleep(1)