How can I use multiprocessing to plot data while sampling sensors?

Question

I am trying to use the multiprocessing python library to plot while collecting data and I do not want to interrupt the data collection or slow it down. I am not sure how to store the data and send it to multiprocessing to cleanly plot. Also not sure if I can use matplotlib.

I have tried to create a pipe and also a queue. Also I have tried using a lock and neither have worked well enough for me. I am not trying to use Pool because I am not running a lot of small processes, I am just running a data collection process and then plotting. From what I understand with playing around with the library, you start a Process and that will get the process to start? Once you join the processes, it will wait until something is returned to then move on in the code. I have also tried my normal data collection and then adding on a separate process for plotting, they would both be continuous loops that would stop with a KeyboardInterrupt.

from multiprocessing import Process, Lock, Pipe
import time
import numpy as np
import matplotlib.pyplot as plt

def collectData(sender):
    xVal = np.random.rand() + np.random.randint(0,10)
    yval = 23 + np.random.rand()
    time.sleep(0.001)
    sender.send([xVal,yVal])

def plottingData(receiver,fig,ax):
    connect = receiver.recv()
    print(connect)
    ax.scatter(connect[0],connect[1])
    plt.show()
if __name__ == '__main__':
    fig, ax = plt.subplots()
    ax.scatter([],[])
    plt.show(block=False)

    receiver, sender = Pipe(False)

    for i in range(10):
        print('Start process...')
        duta = Process(target=collectData, args=(sender,))
        plut = Process(target=plottingData, args=(receiver,fig,ax))

        duta.start()
        plut.start()
        print('...done with process')

    duta.join()
    plut.join()
    print('Completed multiprocessing')   

This is just a simple example I am trying to code to simulate the data collection and plotting by passing data between them. This is a basic layer to try to build upon.

Some things I am trying to do: Loop my plotting like I currently do. It runs continuously and stop with a KeyboardInterrupt. Add on plotting so that I can see the data but I do not want to slow down the data collection.

From what I have understood with the code as of now. The Processes are looking to finish and then move on, making the continuous collection hard. The plotting tells me this too as I have 10 separate figures open and it does not finish until I close the figure windows. I am trying to pass a pandas data frame to then plot and cannot figure out if that will be possible with a pipe or what.

Any help understanding the multiprocessing library will be greatly appreciated!

Thanks

Answer 1

If you want to continuously receive and plot data you may want to use multithreading and matplotlib's FuncAnimation tools. Set up a thread to receive data and store it in some form of list and then in the animation function update the line/scatter plot values to see the changes reflected on your plot. Make sure that the animation is running in the main loop of your program as otherwise this may cause problems.

Answer 2

As for collecting data in a certain process and plotting it in another you could do something like this:

from multiprocessing import Process, Queue
import time
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation

def collectData(communicator):
    while True:

        xval = np.random.rand() + np.random.randint(0,1)
        yval =  np.random.rand()
        communicator.put([xval,yval]) # we use the Queue here to commuicate to the other process. Any process is
        # allowed to put data into or extract data from. So the data collection process simply keeps putting data in.
        time.sleep(1) # not to overload this example ;)

def update(frame, communicator: Queue): # here frame needs to be accepted by the function since this is used in FuncAnimations
    data = communicator.get() # this blocks untill it gets some data
    xdata.append(data[0])
    ydata.append(data[1])
    ln.set_data(xdata, ydata)
    return ln,

if __name__ == '__main__':
    fig, ax = plt.subplots()
    xdata, ydata = [], []
    ln, = plt.plot([], [], 'ro')

    communicator = Queue()
    print('Start process...')
    duta = Process(target=collectData, args=(communicator,))
    duta.start()
    ani = FuncAnimation(fig, update, blit=True, fargs=(communicator,))
    plt.show()
    print('...done with process')
    duta.join()
    print('Completed multiprocessing')

So the idea is that the duta process keeps adding data to the Queue and the main initial process that calls FuncAnimation keeps waiting untill it finds anything to plot.