What is most efficient Python IPC mechanism for plotting real-time serial data?

Question

What is the fastest Python mechanism for getting data read off of a serial port, to a separate process which is plotting that data?

I am plotting eeg data in real-time that I read off of a serial port. The serial port reading and packet unpacking code works fine, in that if I read and store the data, then later plot the stored data, it looks great. Like this:

note: device generates test sine wave for debugging

I am using pyQtGraph for the plotting. Updating the plot in the same process that I read the serial data in is not an option because the slight delay between serial read() calls causes the serial buffer to overflow and bad check-sums ensue. pyQtGraph has provisions for rendering the graph on a separate process, which is great, but the bottle-neck seems to be in the inter-process communication. I have tried various configuration of Pipe() and Queue(), all of which result in laggy, flickering graph updates. So far, the smoothest, most consistent method of getting new values from the serial port to the graph seems to be through shared memory, like so:

from pyqtgraph.Qt import QtGui
import pyqtgraph as pg
from multiprocessing import Process, Array, Value, Pipe
from serial_interface import EEG64Board
from collections import deque

def serialLoop(arr):
    eeg = EEG64Board(port='/dev/ttyACM0')
    eeg.openSerial() 
    eeg.sendTest('1')        #Tells the eeg device to start sending data
    while True:
        data = eeg.readEEG() #Returns an array of the 8 latest values, one per channel
        if data != False:    #Returns False if bad checksum
            val.value = data[7] 

val = Value('d',0.0)
q = deque([],500)

def graphLoop():
    global val,q
    plt = pg.plot(q)
    while True:
        q.append(val.value)
        plt.plot(q,clear=True)
        QtGui.QApplication.processEvents()

serial_proc = Process(target=serialLoop, args=(val,), name='serial_proc')
serial_proc.start()

try:
    while True:
        graphLoop()
except KeyboardInterrupt:
    print('interrupted')

The above code performs real-time plotting by simply pulling the latest value recorded by the serialLoop and appending it to a deque. While the plot updates smoothly, it is only grabbing about 1 in 4 values, as seen in the resulting plot:

So, what multi-process or thread structure would you recommend, and then what form of IPC should be used between them?

Update:

I am receiving 2,000 samples per second. I am thinking that if I update the display at 100 fps and add 20 new samples per frame then I should be good. What is the best Python multithreading mechanism for implementing this?

Answer 1

This may not be the most efficient, but the following code achieves 100 fps for one plot, or 20 fps for 8 plots. The idea is very simple: share an array, index, and lock. Serial fills array and increments index while is has lock, plotting process periodically grabs all of the new values from the array and decrements index, again, under lock.

from pyqtgraph.Qt import QtGui
import pyqtgraph as pg
from multiprocessing import Process, Array, Value, Lock
from serial_interface import EEG64Board
from collections import deque

def serialLoop(arr,idx,lock):
    eeg = EEG64Board(port='/dev/ttyACM0')
    eeg.openSerial() 
    eeg.sendTest('1')        #Tells the eeg device to start sending data
    while True:
        data = eeg.readEEG() #Returns an array of the 8 latest values, one per channel
        if data != False:    #Returns False if bad checksum
            lock.acquire()
            for i in range(8):
                arr[i][idx.value] = data[i] 
            idx.value += 1
            lock.release()
    eeg.sendTest('2') 

arr = [Array('d',range(1024)) for i in range(8)]
idx = Value('i', 0)
q = [deque([],500) for i in range(8)]
iq = deque([],500)
lock = Lock()

lastUpdate = pg.ptime.time()
avgFps = 0.0

def graphLoop():
    global val,q,lock,arr,iq, lastUpdate, avgFps
    win = pg.GraphicsWindow()
    plt = list()
    for i in range(8):
        plt += [win.addPlot(row=(i+1), col=0, colspan=3)]
    #iplt = pg.plot(iq)
    counter = 0
    while True:
        lock.acquire()
        #time.sleep(.01)
        for i in range(idx.value):
            for j in range(8):
                q[j].append(arr[j][i])        
        idx.value = 0
        lock.release()
        for i in range(8):
            plt[i].plot(q[i],clear=True)
        QtGui.QApplication.processEvents()
        counter += 1

        now = pg.ptime.time()
        fps = 1.0 / (now - lastUpdate)
        lastUpdate = now
        avgFps = avgFps * 0.8 + fps * 0.2

serial_proc = Process(target=serialLoop, args=(arr,idx,lock), name='serial_proc')
serial_proc.start()

graphLoop()

serial_proc.terminate()