[英]Running similar program in multiple cores with different variable
I have a program which I want to create N instances of, where the only thing that varies is some hyper parameter $\\beta$. 我有一个程序,我想创建N个实例,其中唯一不同的是一些超参数$ \\ beta $。
In my mind I know I could do this with a bash script, where I call the program N times, each with a different value for $\\beta$, and send each one to the background so that the next one can run: 在我看来,我知道我可以使用bash脚本执行此操作,我将程序调用N次,每次调用$ \\ beta $的值,并将每个值发送到后台,以便下一个可以运行:
#!/bin/bash
nohup python3 test.py 1 >> res.txt &
nohup python3 test.py 2 >> res.txt &
nohup python3 test.py 3 >> res.txt &
nohup python3 test.py 4 >> res.txt &
Maybe I can also do this directly in python, in a cleaner manner. 也许我也可以在python中以更干净的方式直接执行此操作。 My question is, from your experience, what is the cleanest way of achieving this?
我的问题是,根据您的经验,实现这一目标的最简洁方法是什么? Feel free to ask any detail I might have missed.
随意询问我可能错过的任何细节。
For running multiple things in parallel, the thing that comes to my mind is GNU Parallel . 为了并行运行多个东西,我想到的是GNU Parallel 。
So for your example, a dry-run gives this: 所以对于你的例子,干运行给出了:
parallel --dry-run 'nohup python prog.py {} &' ::: {1..4}
Sample Output 样本输出
nohup python prog.py 3 &
nohup python prog.py 2 &
nohup python prog.py 1 &
nohup python prog.py 4 &
In general, you don't want multiple, parallel processes writing to the same file - it makes a mess, so I would name the output file after the parameter: 一般情况下,您不希望多个并行进程写入同一个文件 - 这会弄得一团糟,所以我会在参数后面命名输出文件:
parallel --dry-run 'nohup python prog.py {} > res{}.txt &' ::: {1..4}
You are looking for the subprocess module. 您正在寻找子进程模块。
subprocess.run([process_name, arg1, arg2, argn])
An Example. 一个例子。
import subprocess
subprocess.run(["ls", "-l"])
Also check how to call a subprocess and get the output 还要检查如何调用子进程并获取输出
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.