Multiprocessing with undetermined number of processes

Question

I'm far from being adapt in python and since 3 days I'm trying to figure out how to properly work with multiprocessing, but now I hit a dead and and need some assistance.

Basically what the program is supposed to do, is controlling different segments of an LED strip from multiple (seni-random) inputs at the same time. Therefore I came to the conclusion that I probably need to use multiprocessing.

I've written a module for it using an existing module from Adafruit. (I stripped it down for demonstration)

import time
import RPi.GPIO as GPIO
from multiprocessing import Lock
import Adafruit_WS2801
import Adafruit_GPIO.SPI as SPI

class Pixels(object):
    def __init__(self, pixelCount, spiPort, spiDevice):
        self.l = Lock()

        self.pixels = Adafruit_WS2801.WS2801Pixels(pixelCount, spi=SPI.SpiDev(spiPort, spiDevice), gpio=GPIO)

        # Clear all the pixels to turn them off.
        self.pixels.clear()
        self.pixels.show() 

    def set_color(self, target_pixel, color=(255,0,0)):
        for k in target_pixel:
            self.l.acquire()
            self.pixels.set_pixel(k, Adafruit_WS2801.RGB_to_color( color[0], color[1], color[2] ))
            self.l.release()
        self.l.acquire()
        self.pixels.show()
        self.l.release()

    def blink_color_blank(self, target_pixel, blink_times=1, wait=0.5, color=(255,0,0)):
        for i in range(blink_times):
            self.set_color(target_pixel, color)
            time.sleep(wait)
            self.set_color(target_pixel, (0,0,0))
            time.sleep(wait)

Inside of self.pixels all the information about which LED should have which color is stored. self.pixels.set_pixel() writes the new values to storage. self.pixels.show() actually sends these values to the SPI-Bus.

Now my attempt at multiprocessing starts like this.

from multiprocessing import Process, Manager
from multiprocessing.managers import BaseManager
import LED_WS2801

if __name__ == '__main__':
    BaseManager.register('LedClass', LED_WS2801.Pixels)
    manager = BaseManager()
    manager.start()
    inst = manager.LedClass(10,0,0)

Now my problem arises when I start a process while another is still active.

    p = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
    p.start()
    p = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
    p.start()
    p.join()

This gives me following error:

Process Process-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
File "<string>", line 2, in blink_color_blank
File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod
    kind, result = conn.recv()
EOFError

But when I do something like this, everything is fine.

    p = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
    p.start()
    b = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
    b.start()
    p.join()
    b.join()

But I don't know my final number of processes as they get spawned by external inputs, so I need some way to control a variable number of processes. My idea was to use a list like this:

    jobs = []

    jobs.append(Process(target=inst.set_color, args=([0,1,2],(255,0,255),)))
    jobs[0].start()

But much to my disappointment this returns with another error:

    Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
File "<string>", line 2, in set_color
File "/usr/lib/python2.7/multiprocessing/managers.py", line 755, in _callmethod
    self._connect()
File "/usr/lib/python2.7/multiprocessing/managers.py", line 742, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 308, in SocketClient
    s.connect(address)
File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory

I hope I made my problem as understandable and clear as possible. As I haven't found anything like this I guess I'm doing something fundamentally wrong. So would you help me out, please? Thank you.

Answer 1

you have to wait all child processes to finish its job, with re-asign p :

p = Process(...)
p.start()
p = Process(...)
p.start()
p.join()

you are just waiting for the later one in p to finish, the error comes when master wants to terminate but the first child process is still running. try this to wait for all child to finish:

p1 = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
p1.start()
p2 = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
p2.start()
childs = [p1, p2]
while any(p.is_alive() for p in childs):
    for p in childs:
        p.join(1)

besides, there is an multiprocessing.active_children() api to get all the children of the current process, in case you really cant gather the list from the beginning.