Python run to run Batch Command in Parralel

Question

I have a List which has all my Batch Command.(More than 1000) count I need to run the commands from this list 5 at a time(in Parallel), If any one completes the 6th should kick off.

Can you please assist? Thanks

Answer 1

If you don't need output from the Batch commands, you can simply

import subprocess
subprocess.Popen("command string here", shell=True) 

This will run your batch code in the shell attached to this Python runtime.

To run things in parallel, you can just track how many are currently running. I like to use threads for this

import subprocess
from threading import Thread
import time

processing = 0

def call_batch(command):
    global processing
    process = subprocess.Popen("command string here", shell=True)
    process.wait()
    processing -= 1

if __name__ == "__main__":
    commands = []
    ##load commands

    for command in commands:
        if processing < 5:
            t = Thread(target=call_batch, args=(command))
            t.daemon = True
            t.start()
            processing += 1
        else:
            time.sleep(0.1) # I don't know how long you expect these commands to take so this is allowing a max of 5 * (1/0.1) = 50 per second

If you come from another programming background, you will notice the lack of locks. This is because of the global interpreter lock.

If you know a lot about Python, you will notice my recommendation to use shell=True . I recommend it because it is simple and not dangerous when executed on trusted input however the OP should decide whether to use shell=True based on the scenario.

Reading on Thread : https://docs.python.org/2/library/threading.html#thread-objects
Reading on subprocess : https://docs.python.org/2/library/subprocess.html
Docs on why shell=True is dangerous : https://docs.python.org/2/library/subprocess.html#frequently-used-arguments

You may like to use the ThreadPool from the multiprocessing library if you don't need a lot of control over the system. See http://chriskiehl.com/article/parallelism-in-one-line/ about half-way down for an example of multiprocessing.ThreadPool.map for mapping a function to a number of threads.