Get python subprocess run time

Question

I am using python to run multiple subprocesses at the same time.

I want to get the run time of each process.

I am using the subprocess module.

What I did: I created two separate for loops: The first one for running each process The second waits for all processes to end.

for prcs in batch:
    p = subprocess.Popen([prcs])
    ps.append(p)
for p in ps:
    p.wait()

This code works fine for running the processes simultaneously, but I do not know what to add to it in order to get the run time of each process separately.

Edit: Is there a way to get the run time through the module subprocess? For example: runtime = p.runtime()

Answer 1

I agree with @quamrana that the easiest way to do this would be with threads.

First, we need to import some standard library modules:

import collections
import subprocess
import threading
import time

Instead of a list to store the processes, we use an ordered dictionary to keep track of the processes and their times. Since we don't know how long each thread will take, we need some way to keep track of the original order of our {process: time} pairs. The threads themselves can be stored in a list.

ps = collections.OrderedDict()
ts = []

Initializing the value paired to each process as the current time makes the whole thing cleaner, despite the fact that it is generally inadvisable to use the same variable for two different things (in this case, starting time followed by process duration). The target for our thread simply waits for the thread to finish and updates the ps ordered dictionary from the start time to the process duration.

def time_p(p):
    p.wait()
    ps[p] = time.time() - ps[p]

for prcs in batch:
    p = subprocess.Popen([prcs])
    ps[p] = time.time()
    ts.append(threading.Thread(target=time_p, args=(p,)))

Now, we just start each of the threads, then wait for them all to complete.

for t in ts:
    t.start()

for t in ts:
    t.join()

Once they are all complete, we can print out the results for each:

for prcs, p in zip(batch, ps):
    print('%s took %s seconds' % (prcs, ps[p]))