Non-blocking subprocess and catch the output of each subprocess

Question

so I got this tricky situation which I need to execute few subprocess and be able to get each subprocess output. This is what I'm using atm:

output = Popen(cmd, stdout=PIPE, stderr=STDOUT)
output = output.communicate()[0]

Which works really well and also getting me errors, if there are any. Now I need to execute like 10 of those and be able to get the output of each one of them. I was wondering if there is a smooth way of doing that WITHOUT using multithreading.

Answer 1

subprocess.Popen starts the child process asynchronously, only communicate makes the operation synchronous.

So you just have to first launch all your children processes. Then depending on how you want to process their output, you can:

• if you need to process output in a ordered manner, that is display output from cmd1, then cmd2, ... just use communicate or wait for each child command
• if you need to process output as soon as it is available and you use a Linux or Unix-like platform, you could use the select module.
• if you need to process output as soon as each command ends and want to support Windows platforms, you could just loop with subprocess.poll on child processes to know which have finished.
• if you need more, you are likely to have to use threading