Control the amount of jobs in multiprocessing async python

Question

I am trying to make a script that will use only 4 process at a time and start another one once a value is returned. I think some of the problem is that results.get keeps waiting until it gets a results and will not continue until a value is returned. I want the While Loop to continue while I am waiting for the result.

https://docs.python.org/2.7/library/multiprocessing.html#multiprocessing.pool.AsyncResult.get

import multiprocessing as mp
import time
from random import randint


def print_hello(VPN_Name):
    time.sleep(randint(0,5))
    return VPN_Name


VPN_list = ['3048-VPN01', '3049-VPN01', '3051-VPN01', '3053-VPN01', '3058-VPN01', '3059-VPN01', '3061-MULTI01', '3063-VPN01', '3065-VPN01', '3066-MULTI01', '3067-VPN01', '3069-VPN01', '3071-VPN01', '3072-VPN01']

VPN_len = len(VPN_list)
x = 0
pool = mp.Pool(processes=4)

job_tracker = []
complete_tracker = []


while True:
    for VPN_Name in VPN_list:
        if VPN_len == 0:
            break
        while True:
            print "Complete Job Tracker ", complete_tracker
            print "Job Tracker ", job_tracker
            for comp in complete_tracker:
                if comp in job_tracker:
                    x = x - 1
                    job_tracker.remove(comp)
                    print "Confirmed complete " + comp
                continue
            if x < 4:
                results = pool.apply_async(print_hello,args=(VPN_Name,))
                VPN_len = VPN_len - 1
                x = x + 1
                print "Started  " + VPN_Name
                job_tracker.append(VPN_Name)
                complete_tracker.append(results.get())
                break
            continue

Answer 1

Your loop does not work because results.get blocks until the result is available, which effectively makes your code non-parallel. You seem to be trying to do a lot of extra work to get functionality that multiprocessing.Pool provides for you automatically.

When you do pm.Pool(4) , you create a pool of 4 processes, so when you pass many tasks to the pool it will execute 4 at a time until they are all complete. There are even functions to submit a collection of inputs to the pool so you don't have to iterate yourself.

This lets you replace your whole while loop with:

pool = mp.Pool(processes=4)

results = pool.map(print_hello, VPN_list)

for result in results:
    print "Confirmed complete " + result

This will block at pool.map until all the tasks are complete and then return them all at once in the order that you submitted them. If you want them to return as they are completed (but still in order), you can use pool.imap , and if you don't care about order but just want results as soon as they are available, use pool.imap_unordered