Run multiple bash commands simultaneously in Python

Question

I want to create a python script that can start multiple instances of a bash script.

I have tried

import subprocess

    commands = '''
    bashcmd1
    bashcmd2
    bashcmd3
    '''

    process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    out, err = process.communicate(commands)
    print out

But it stops at bashcmd2 and continues only if i hit CTRL+C .

Answer 1

you're just running one bash with 3 commands in it.

If the commands aren't setting variables or depending from each other (else you would not be able to parallelize them), maybe you could create 3 subprocess.Popen instances instead:

commands = '''
bashcmd1
bashcmd2
bashcmd3
'''

for process in [subprocess.Popen(['/bin/bash', '-c', line], stdout=subprocess.PIPE) 
                for line in commands.split("\n") if line]:  # filter out blank lines
    out, err = process.communicate() # or just rc = process.wait()
    # print out & err

that command first create a list comprehension of Popen objects (list not generator so the processes start immediately), then perform a communicate to wait for completion (but other processes are running in the meanwhile)

The upside is that you can apply this technique to any script containing commands, and you don't need to use the shell & capability (more portable, including Windows provided you're using ["cmd","/c" prefix instead of bash )