How to optimise plotting Serial data in real-time using Python

Question

I'm trying to plot tab separated values I receive from a serial device in real-time. I'm pretty new to python but have managed to cobble together a script that manages it however it can't seem to handle the rate at which data is received and uses a lot of processing power before slowing and eventually freezing. Is the anything I can do prevent this. I've attached an example of the data I'm working with and my script

The data I receive looks like this and is received at a rate of about a line every half a second.

546     5986637 3598844 +26.0   01A0
547     5986641 3598843 +25.50  0198
548     5986634 3598844 +24.50  0188

from matplotlib import pyplot as plt
from matplotlib import animation
import serial
from pandas import DataFrame
from datetime import datetime
import csv

filename = datetime.now().strftime("%d-%m-%Y_%I-%M-%S_%p")  # Gets time and date in readable format for filenaming.
Data1 = {'Value': [0], 'Frequency 1': [0], 'Frequency2': [0], 'Temperature': [0]}
df = DataFrame(Data1, columns=['Value', 'Frequency1', 'Frequency2', 'Temperature'])
serial_port = 'COM5';  # Different port for linux/mac
baud_rate = 9600;  # In arduino, Serial.begin(baud_rate)
write_to_file_path = "output.txt";
data = []
ft = []
output_file = open(write_to_file_path, "w+");
ser = serial.Serial(serial_port, baud_rate)

plt.ion()
fig, (ax1, ax2, ax3) = plt.subplots(3, 1, sharex=True, sharey=False, )

ax1.set_title('Temp')
ax2.set_title('Freq 1')
ax3.set_title('Freq 2')
ax1.set_ylabel('Temperature')
ax2.set_ylabel('Frequency')
ax3.set_ylabel('Frequency')
ax1.ticklabel_format(useOffset=False)
ax2.ticklabel_format(useOffset=False)
ax3.ticklabel_format(useOffset=False)
ax1.ticklabel_format(style='plain', axis='y', scilimits=(0, 0))
ax2.ticklabel_format(style='sci', axis='y', scilimits=(6, 6))
ax3.ticklabel_format(style='sci', axis='y', scilimits=(6, 6))
while True:
    line = ser.readline();
    line = line.decode("utf-8")  # ser.readline returns a binary, convert to string
    print(line)
    line1 = line.split('\t')  # Separates values by tabs
    output_file.write(line);  # Writes to text file
    data.append(line1)  # Adds line to data file
    newline = [float(line1[0]), float(line1[1]), float(line1[2]), float(line1[3])]  # Creates line of float values
    ft.append(newline)  # Adds to list of floats
    f1 = float(line1[0])  # Line number (count)
    f2 = float(line1[1])  # Frequency 1
    f3 = float(line1[2])  # Frequency 2
    f4 = float(line1[3])  # Temperature in C
    f5 = str(line1[4])  # Temperature in Hex, treated as a string
    #    Data2 = {'Value':[f1],'Frequency 1':[f2],'Frequency2':[f3], 'Temperature':[f4]}
    #    df2 = DataFrame(Data2,columns=['Value', 'Frequency1','Frequency2','Temperature'])
    #    df.append(df2)

    # DataFrame still not working, need to fix so that data is stores as integer or float
    plt.pause(0.1)

    ax1.plot(f1, f4, marker='.', linestyle='solid')  # subplot of freq 1
    ax2.plot(f1, f2, marker='.', linestyle='solid')  # subplot of freq 2
    ax3.plot(f1, f3, marker='.', linestyle='solid')  # subplot of Temp in C
    plt.subplot
    plt.xlabel("Count")
    with open(filename + ".csv", "a") as f:  # Writes data to CSV, hex values for temp don't seem to be writing
        writer = csv.writer(f, delimiter=",")
        writer.writerow([f1, f2, f3, f4, f5])

    plt.draw()
    plt.savefig(filename + '.png', bbox_inches='tight')  # Saves the plot

Answer 1

You may consider using threads to split up your tasks. You probably don't need to save the figure every time you recieve new data. You can reduce your computational load by only updating the plot every 30 seconds or so for example. You can also split out writing to the csv, that way you have three threads, one looking for data, one storing buffered data, and one updating your plot.

This answer may be a good reference.

At the end of foo(), create a Timer which calls foo() itself after 10 seconds. Because, Timer create a new thread to call foo().

import time, threading
def foo():
    print(time.ctime())
    threading.Timer(10, foo).start()

foo()

#output:
#Thu Dec 22 14:46:08 2011
#Thu Dec 22 14:46:18 2011
#Thu Dec 22 14:46:28 2011
#Thu Dec 22 14:46:38 2011