Python3 Pool async processes | workers

Question

I am trying to use 4 processes for 4 async methods.

Here is my code for 1 async method (x):

from multiprocessing import Pool
import time

def x(i):
     while(i < 100):
          print(i)
          i += 1
          time.sleep(1)

def finish(str):
     print("done!")

if __name__ == "__main__":
     pool = Pool(processes=5)
     result = pool.apply_async(x, [0], callback=finish)

print("start")

according to: https://docs.python.org/2/library/multiprocessing.html#multiprocessing.JoinableQueue the parameter processes in Pool is the number of workers.

How can i use each of these workers?

EDIT: my ASYNC class

from multiprocessing import Pool
import time

class ASYNC(object):
    def __init__(self, THREADS=[]):
        print('do')
        pool = Pool(processes=len(THREADS))
        self.THREAD_POOL = {}
        thread_index = 0
        for thread_ in THREADS:
            self.THREAD_POOL[thread_index] = {
                'thread': thread_['thread'],
                'args': thread_['args'],
                'callback': thread_['callback']
            }
            pool.apply_async(self.run, [thread_index], callback=thread_['callback'])
            self.THREAD_POOL[thread_index]['running'] = True
            thread_index += 1
    def run(self, thread_index):
        print('enter')
        while(self.THREAD_POOL[thread_index]['running']):
            print("loop")
            self.THREAD_POOL[thread_index]['thread'](self.THREAD_POOL[thread_index])
            time.sleep(1)
        self.THREAD_POOL[thread_index]['running'] = False
    def wait_for_finish(self):
        for pool in self.THREAD_POOL:
            while(self.THREAD_POOL[pool]['running']):
                time.sleep(1)
def x(pool):
    print(str(pool))
    pool['args'][0] += 1


def y(str):
    print("done")

A = ASYNC([{'thread': x, 'args':[10], 'callback':y}])

print("start")
A.wait_for_finish()

Answer 1

multiprocessing.Pool is designed to be a convenient way of distributing work to a pool of workers, without worrying about which worker does which work. The reason that it has a size is to allow you to be lazy about how quickly you dispatch work to the queue and to limit the expensive (relatively) overhead of creating child processes.

So the answer to your question is in principle you shouldn't be able to access individual workers in a Pool. If you want to be able to address workers individually, you will need to implement your own work distribution system and using multiprocessing.Process , something like:

from multiprocessing import Process

def x(i):
    while(i < 100):
        print(i)
        i += 1

pools = [Process(target=x, args=(1,)) for _ in range(5)]
map(lambda pool: pool.start(), pools)
map(lambda pool: pool.join(), pools)
print('Done!')

And now you can access each worker directly. If you want to be able to send work dynamically to each worker while it's running (not just give it one thing to do like I did in my example) then you'll have to implement that yourself, potentially using multiprocessing.Queue . Have a look at the code for multiprocessing to see how that distributes work to its workers to get an idea of how to do this.

Why do you want to do this anyway? If it's just concern about whether the workers get scheduled efficiently, then my advice would just be to trust multiprocessing to get that right for you, unless you have really good evidence that in your case it does not for some reason.