使用 Python 连续读取和绘制 CSV 文件

Question

this is my first time asking something here, so I hope I am asking the following question the "correct way". If not, please let me know, and I will give more information.

I am using one Python script, to read and write 4000Hz of serial data to a CSV file.

The structure of the CSV file is as follows: (this example shows the beginning of the file)

Time of mSure Calibration: 24.10.2020 20:03:14.462654
Calibration Data - AICC: 833.95; AICERT: 2109; AVCC: 0.00; AVCERT: 0 
Sampling Frequency: 4000Hz
timestamp,instantaneousCurrentValue,instantaneousVoltageValue,activePowerValueCalculated,activePowerValue
24.10.2020 20:03:16.495828,-0.00032,7e-05,-0.0,0.0
24.10.2020 20:03:16.496078,0.001424,7e-05,0.0,0.0
24.10.2020 20:03:16.496328,9.6e-05,7e-05,0.0,0.0
24.10.2020 20:03:16.496578,-0.000912,7e-05,-0.0,0.0

Data will be written to this CSV as long as the script reading serial data is active. Thus, this might become a huge file at some time. (Data is written in chunks of 8000 rows = every two seconds)

Here is my problem: I want to plot this data live. For example, update the plot each time data is written to the CSV file. The plotting shall be done from another script than the script reading and writing the serial data.

What is working: 1. Creating the CSV file. 2. Plotting a finished CSV file using another script - actually pretty well :-)

I have this script for plotting:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

"""Data Computation Software for TeensyDAQ - Reads and computes CSV-File"""

# region imports
import getopt
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import pathlib
from scipy.signal import argrelextrema
import sys
# endregion

# region globals
inputfile = ''
outputfile = ''
# endregion

# region functions
def main(argv):
    """Main application"""

    # region define variables
    global inputfile
    global outputfile
    inputfile = str(pathlib.Path(__file__).parent.absolute(
    ).resolve())+"\\noFilenameProvided.csv"
    outputfile = str(pathlib.Path(__file__).parent.absolute(
    ).resolve())+"\\noFilenameProvidedOut.csv"
    # endregion

    # region read system arguments
    try:
        opts, args = getopt.getopt(
            argv, "hi:o:", ["infile=", "outfile="])
    except getopt.GetoptError:
        print('dataComputation.py -i <inputfile> -o <outputfile>')
        sys.exit(2)
    for opt, arg in opts:
        if opt == '-h':
            print('dataComputation.py -i <inputfile> -o <outputfile>')
            sys.exit()
        elif opt in ("-i", "--infile"):
            inputfile = str(pathlib.Path(
                __file__).parent.absolute().resolve())+"\\"+arg
        elif opt in ("-o", "--outfile"):
            outputfile = str(pathlib.Path(
                __file__).parent.absolute().resolve())+"\\"+arg
    # endregion

    # region read csv
    colTypes = {'timestamp': 'str',
                'instantaneousCurrent': 'float',
                'instantaneousVoltage': 'float',
                'activePowerCalculated': 'float',
                'activePower': 'float',
                'apparentPower': 'float',
                'fundReactivePower': 'float'
                }
    cols = list(colTypes.keys())
    df = pd.read_csv(inputfile, usecols=cols, dtype=colTypes,
                     parse_dates=True, dayfirst=True, skiprows=3)
    df['timestamp'] = pd.to_datetime(
        df['timestamp'], utc=True, format='%d.%m.%Y %H:%M:%S.%f')
    df.insert(loc=0, column='tick', value=np.arange(len(df)))
    # endregion

    # region plot data
    fig, axes = plt.subplots(nrows=6, ncols=1,  sharex=True, figsize=(16,8))
    fig.canvas.set_window_title(df['timestamp'].iloc[0]) 
    fig.align_ylabels(axes[0:5])

    df['instantaneousCurrent'].plot(ax=axes[0], color='red'); axes[0].set_title('Momentanstrom'); axes[0].set_ylabel('A',rotation=0)
    df['instantaneousVoltage'].plot(ax=axes[1], color='blue'); axes[1].set_title('Momentanspannung'); axes[1].set_ylabel('V',rotation=0)
    df['activePowerCalculated'].plot(ax=axes[2], color='green'); axes[2].set_title('Momentanleistung ungefiltert'); axes[2].set_ylabel('W',rotation=0)
    df['activePower'].plot(ax=axes[3], color='brown'); axes[3].set_title('Momentanleistung'); axes[3].set_ylabel('W',rotation=0)
    df['apparentPower'].plot(ax=axes[4], color='brown'); axes[4].set_title('Scheinleistung'); axes[4].set_ylabel('VA',rotation=0)
    df['fundReactivePower'].plot(ax=axes[5], color='brown'); axes[5].set_title('Blindleitsung'); axes[5].set_ylabel('VAr',rotation=0); axes[5].set_xlabel('microseconds since start')
    
    plt.tight_layout()    
    plt.show()
    # endregion

# endregion


if __name__ == "__main__":
    main(sys.argv[1:])

My thoughts on how to solve my problem:

1. Modify my plotting script to continuously read the CSV file and plot using the animation function of matplotlib.
2. Using some sort of streaming functionality to read the CSV in a stream. I have read about the streamz library, but I have no idea how I could use it.

Any help is highly appreciated!

Kind regards, Sascha

EDIT 31.10.2020:

Since I am not aware of the mean duration, how long to wait for help, I try to add more input, which maybe leads to helpful comments.

I wrote this script to write data continuously to a CSV file, which emulates my real script without the need for external hardware: (Random data is produced and CSV-formatted using a timer. Each time there are 50 new rows, the data is written to a CSV file)

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import csv
from random import randrange
import time
import threading
import pathlib
from datetime import datetime, timedelta

datarows = list()
datarowsToWrite = list()
outputfile = str(pathlib.Path(__file__).parent.absolute().resolve()) + "\\noFilenameProvided.csv"
sampleCount = 0

def startBatchWriteThread():
    global outputfile
    global datarows
    global datarowsToWrite
    datarowsToWrite.clear()
    datarowsToWrite = datarows[:]
    datarows.clear()
    thread = threading.Thread(target=batchWriteData,args=(outputfile, datarowsToWrite))
    thread.start()

def batchWriteData(file, data):
    print("Items to write: " + str(len(data)))
    with open(file, 'a+') as f:
        for item in data:
            f.write("%s\n" % item)

def generateDatarows():
    global sampleCount
    timer1 = threading.Timer(0.001, generateDatarows)
    timer1.daemon = True
    timer1.start()
    datarow = datetime.now().strftime("%d.%m.%Y %H:%M:%S.%f")[:] + "," + str(randrange(10)) + "," + str(randrange(10)) + "," + str(randrange(10)) + "," + str(randrange(10)) + "," + str(randrange(10)) + "," + str(randrange(10))
    datarows.append(datarow)
    sampleCount += 1

try:
    datarows.append("row 1")
    datarows.append("row 2")
    datarows.append("row 3")
    datarows.append("timestamp,instantaneousCurrent,instantaneousVoltage,activePowerCalculated,activePower,apparentPower,fundReactivePower")
    startBatchWriteThread()
    generateDatarows()
    while True:
        if len(datarows) == 50:
            startBatchWriteThread()
except KeyboardInterrupt:
    print("Shutting down, writing the rest of the buffer.")
    batchWriteData(outputfile, datarows)
    print("Done, writing " + outputfile)

The script from my initial post can then plot the data from the CSV file.

I need to plot the data as it is written to the CSV file to see the data more or less live.

Hope this makes my problem more understandable.

Answer 1

For the Googlers: I could not find a way to achieve my goal as described in the question.

However, if you are trying to plot live data, coming with high speed over serial comms (4000Hz in my case), I recommend designing your application as a single program with multiple processes.

The problem in my special case was, that when I tried to plot and compute the incoming data simultaneously in the same thread/task/process/whatever, my serial receive rate went down to 100Hz instead of 4kHz. The solution with multiprocessing and passing data using the quick_queue module between the processes I could resolve the problem.

I ended up, having a program, which receives data from a Teensy via serial communication at 4kHz, this incoming data was buffered to blocks of 4000 samples and then the data was pushed to the plotting process and additionally, the block was written to a CSV-file in a separate Thread.

Best, S