如何在 for 循环中使用 subprocess.Popen？

Question

我正在尝试使用 subprocess.Popen 来运行命令，我选择了 subprocess.run，因为我需要将命令的输出打印到 shell。

我在 for 循环中调用命令，并为每次迭代修改环境，但 python 只运行第一个命令，而不是其他 6 个。

我很确定这是一个错误，因为我找不到其他类似的东西，但我想我会在提交“错误”之前进行检查。

相关代码如下所示

input_file = Path("input.json").open()

for thread in threads:
    new_env = default_env
    new_env["OMP_NUM_THREADS"] = str(thread)
    print("Starting run for {} threads".format(thread))
    process = subprocess.Popen(
        benchmark_command, env=new_env, stdin=input_file, stdout=subprocess.PIPE)
    lines = []
    for line in process.stdout:
        print(line.decode(), end='')
    results.append(Result(lines, thread))
    print("Completed run for {} threads".format(thread))

现在的输出是这个

❯ python python/plot_results.py
Starting run for 1 threads
<expected output from running command>
Completed run for 1 threads
Starting run for 2 threads
Completed run for 2 threads
Starting run for 4 threads
Completed run for 4 threads
Starting run for 8 threads
Completed run for 8 threads
Starting run for 16 threads
Completed run for 16 threads
Starting run for 32 threads
Completed run for 32 threads
Starting run for 56 threads
Completed run for 56 threads

但应该看起来像这样

❯ python python/plot_results.py
Starting run for 1 threads
<expected output from running command>
Completed run for 1 threads
Starting run for 2 threads
<expected output from running command>
Completed run for 2 threads
Starting run for 4 threads
<expected output from running command>
Completed run for 4 threads
Starting run for 8 threads
<expected output from running command>
Completed run for 8 threads
Starting run for 16 threads
<expected output from running command>
Completed run for 16 threads
Starting run for 32 threads
<expected output from running command>
Completed run for 32 threads
Starting run for 56 threads
<expected output from running command>
Completed run for 56 threads

Answer 1

首先我想说，除非你在语言方面的专业水平远远超过高级，或者你的问题的性质非常罕见，否则你的问题不太可能是一个错误，相反，它可能是你的一些误解关于语言如何做事。

在您的情况下， Popen不会阻止您的主线程。 您的问题是主线程正在启​​动所有进程......并最终开始从标准输出读取输出并打印它。

简单的解决方案是使用subprocess.check_output来等待命令完成，但它不允许env参数，因此我们将在调用 Popen 后使用communicate()方法，这也会阻塞您的主线程并等待终止过程：

for thread in threads:
    new_env = default_env
    new_env["OMP_NUM_THREADS"] = str(thread)
    print("Starting run for {} threads".format(thread))
    output = subprocess.Popen(
        benchmark_command, env=new_env, stdin=input_file, stdout=subprocess.PIPE)
    stdout, stderr = output.communicate()
    lines = []
    for line in stdout:
        print(line.decode(), end='')
    results.append(Result(lines, thread))
    print("Completed run for {} threads".format(thread))

Answer 2

根据@jasonharper 的建议，我需要为每次迭代创建一个新的打开文件对象。

input_file = Path("input.json")

for thread in threads:
    new_env = default_env
    new_env["OMP_NUM_THREADS"] = str(thread)
    print("Starting run for {} threads".format(thread))
    process = subprocess.Popen(
        benchmark_command, env=new_env, stdin=input_file.open(), stdout=subprocess.PIPE)
    lines = []
    for line in process.stdout:
        print(line.decode(), end='')
    results.append(Result(lines, thread))
    print("Completed run for {} threads".format(thread))