Streaming Grid Display in Jupyter Notebook

Question

I am trying to display live price updates coming from a redis pubsub channel in a grid in Jupyter. Everytime there is a price update, the message will be added at the end of the grid. In order words, a gridview widget will be tied to a Dataframe so everytime it changes, the gridview will change. The idea is to get something like this:

I tried to do that by displaying and clearing the output. However, I am not getting a the streaming grid that gets updated in-place but rather displaying and clearing the output which is very annoying.

Here is the output widget in one jupyter cell

import ipywidgets as iw
from IPython.display import display 

o = iw.Output()
def output_to_widget(df, output_widget): 
    output_widget.clear_output()
    with output_widget: 
        display(df)
o

Here is the code to subscribe to redis and get handle the message

import redis, json, time

r = redis.StrictRedis(host = HOST, password = PASS, port = PORT, db = DB)
p = r.pubsub(ignore_subscribe_messages=True)
p.subscribe('QUOTES')

mdf = pd.DataFrame()
while True:
    message = p.get_message()
    if message:
        json_msg = json.loads(message['data'])
        df = pd.DataFrame([json_msg]).set_index('sym')
        mdf = mdf.append(df)
        output_to_widget(mdf, o)
    time.sleep(0.001)

Answer 1

Try changing the first line of output_to_widget to output_widget.clear_output(wait = True) .

https://ipython.org/ipython-doc/3/api/generated/IPython.display.html

Answer 2

I was able to get it to work using Streaming DataFrames from the streamz library.

Here is the class to emit the data to the streamming dataframe.

class DataEmitter:
def __init__(self, pubsub, src):
    self.pubsub = pubsub
    self.src = src
    self.thread = None

def emit_data(self, channel):
    self.pubsub.subscribe(**{channel: self._handler})
    self.thread = self.pubsub.run_in_thread(sleep_time=0.001)

def stop(self):
    self.pubsub.unsubscribe()
    self.thread.stop()    

def _handler(self, message):
    json_msg = json.loads(message['data'])
    df = pd.DataFrame([json_msg])
    self.src.emit(df)

and here is the cell to display the streaming dataframe

r = redis.StrictRedis(host = HOST, password = PASS, port = PORT, db = DB)
p = r.pubsub(ignore_subscribe_messages=True)
source = Stream()
emitter = DataEmitter(p, source, COLUMNS)
emitter.emit_data(src='PRICE_UPDATES')

#sample for how the dataframe it's going to look like
example = pd.DataFrame({'time': [], 'sym': []})
sdf = source.to_dataframe(example=example)
sdf