Python: Using Multiprocess to switch functions

Question

I'm trying to learn about multiprocessing in python.

What I want to happen is have x increase with time, then when I hit 'enter' it will decrease with time and I can keep hitting enter to switch from increasing to decreasing.

All i've managed to make so far is this

import time
from multiprocessing import Pool

def f(x):
    return (x+1)
def f1(x):
    return (x-1)

if __name__ == '__main__':
    pool = Pool(processes=2)
    x=0
    while x<100:
        x = pool.apply_async(f, [10])
        time.sleep(0.05)

I'm having trouble looking though the documentation because they don't give explicit examples. Please help I have no idea

Answer 1

Threw together an example to give you some ideas.

#!/usr/bin/python3
import time
from multiprocessing import Pool, Manager
from threading import Thread

NUM_PROCESSES = 4

def consumer(q):
    out = []
    while True:
        val = q.get()
        if val is None:
            #poison pill
            break
        #doing some work here
        time.sleep(0.01)
        out.append(val)
    return out

def producer(queue):
    flip = True
    val = 5

    def flipper():
        nonlocal flip
        input('enter to flip it!')
        while True:
            flip = not flip
            txt = 'up' if flip else 'down'
            input('val is {}, now counting {}'.format(val, txt))

    t = Thread(target=flipper, args=(), daemon=True)
    t.start()

    while val > 0:
        for _ in range(NUM_PROCESSES):
            queue.put(val)
        val = val + (1 if flip else -1)
        time.sleep(0.2)

    print()
    print('hit zero, shutting down.')

    for _ in range(NUM_PROCESSES):
        #poison pills
        queue.put(None)

if __name__ == '__main__':
    pool = Pool(processes=NUM_PROCESSES)
    m = Manager() # N.B.: multiprocessing.Queue doesn't work with Pool.apply_async
    q = m.Queue()

    results = pool.apply_async(consumer, args=(q,))
    producer(q) # running in main thread because I'm lazy

    print(results.get())

output:

ben@nixbox:~$ python3 multithread.py 
enter to flip it!
val is 13, now counting down
val is 4, now counting up
val is 8, now counting down
hit zero, shutting down.
[5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 11, 11, 11, 11, 12, 12, 12, 12, 13, 13, 13, 13, 12, 12, 12, 12, 11, 11, 11, 11, 10, 10, 10, 10, 9, 9, 9, 9, 8, 8, 8, 8, 7, 7, 7, 7, 6, 6, 6, 6, 5, 5, 5, 5, 4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 8, 8, 8, 8, 7, 7, 7, 7, 6, 6, 6, 6, 5, 5, 5, 5, 4, 4, 4, 4, 3, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 1]

The way your original code was set up, parallelizing didn't make a whole lot of sense. So I switched to a producer/consumer paradigm: your producer produces "work units" that increase until you hit enter, then decrease until it hits zero, then it loads up the work queue with poison pills that tell the consumers to shut down. There's a few tricks to get the input/flipping thing to work, but I'll just let you look at the code for that. Note that I used threads to wait on blocking I/O operations and processes to do (simulated) CPU-heavy work. This is on purpose.

There are lots of ways to structure multiprocessing code but I find producer/consumer to be a very useful pattern. One thing you really don't want to be doing is rapidly creating processes to do a teeny tiny amount of work and then tearing them down. This is slow ! (I will note that using Pool correctly avoids this - it keeps its workers alive for you. Still, worth mentioning since a lot of people start out by firing off a bunch of Process es in a loop, then later wonder why their parallel code is slower than their single-threaded code :-))

Answer 2

This isn't really a great way to test multiprocessing . You're doing an operation to x that can't really effectively be parralellized - you only want one process to be able to increment x at a time. Otherwise, different workers will stomp on the values of x being set by other workers. And, since all your workers are doing is incrementing x, they're basically going to need to run completely sequentially, with no concurrency at all.

That said, for this code to work properly, you need to actually fetch the real result from the AsyncResult object returned by apply_async :

import time
from multiprocessing import Pool

def f(x):
    return (x+1)
def f1(x1):
    return (x-1)

if __name__ == '__main__':
    pool = Pool(processes=2)
    x=0
    while x<100:
        x = pool.apply_async(f, [x]).get()  # Added get, changed 10 to x
        time.sleep(0.05)

Adding the get call will make the main process block until the worker has actually completed. Of course, this means you're not really doing anything in parrallel, but that's really the only way to increment the variable safely.