在Python脚本中，运行目录中的所有Python脚本并流输出

Question

My file structure looks like this:

runner.py
scripts/
    something_a/
        main.py
        other_file.py
    something_b/
        main.py
        anythingelse.py
    something_c/
        main.py
    ...

runner.py should look at all folders in scripts/ in run the main.py located there.

Right now I'm achieving this through subprocess.check_output . It works, but some of these scripts take a long time to run and I don't get to see any progress; it prints everything after the process has finished.

I'm hoping to find a solution that allows for 2 things to be done somewhat easily:

1) Stream the output instead of getting it all at the end

2) Doesn't prohibit running multiple scripts at once

Is this possible? A lot of the solutions I've seen for running a Python script from another require knowledge of the other script's name/location. I can also enforce that all the main.py 's have a specific function if that helps.

Answer 1

You could use Popen to loop through each file and write its content to multiple log files. Then, you could read from these files in real-time, while each one is populated. :)

How you would want to translate the output to a more readable format, is a little bit more tricky because of readability. You could create another script which reads these log files, with Popen, and decide on how you'd like this information read back in a understandable manner.

""" Use the same command as you would do for check_output """
cmd = ''

for filename in scriptList:
   log = filename + ".log"
   with io.open(filename, mode=log) as out:
        subprocess.Popen(cmd, stdout=out, stderr=out)