[英]Sending command output to a unique file using xargs in Bash
I am trying to run a number of processes dynamically on a fixed number of processors. 我正在尝试在固定数量的处理器上动态运行多个进程。 I want to print the output to a unique file for each process but there is some problem with xargs not using the in-place filename to create a separate file for each process.
我想将输出打印到每个进程的唯一文件中,但是xargs不能使用就地文件名为每个进程创建单独的文件存在一些问题。
The Bash script calls a Csh script and is below: Bash脚本调用Csh脚本,如下所示:
$ cat temp | xargs -P 8 % csh '%'.csh >& '%'.log
Where temp is a text file of csh command names. 其中temp是csh命令名称的文本文件。
My problem is that xargs takes %.log
literally and constantly overwrites the file as the processes write to it, rather than having seperate .log
files as desired. 我的问题是xargs从字面上接受
%.log
并在进程写入文件时不断覆盖该文件,而不是根据需要具有单独的.log
文件。
I run this script as $ bash run.bash &
我以
$ bash run.bash &
运行该脚本
In general, using string replacement to substitute into code is a Bad Idea -- in your case, if you had a script with a malicious name, that name could be used to run arbitrary commands. 通常,使用字符串替换来替换为代码是一个坏主意-在您的情况下,如果您的脚本带有恶意名称,则可以使用该名称来运行任意命令。 (Sure, you're executing the script, but the same would apply in situations where you were purely dealing with data and output file names -- so it's best to make a habit of the robust approach regardless).
(当然,您正在执行脚本,但是在您纯粹处理数据和输出文件名的情况下也是如此—因此,无论如何,最好养成健壮的方法)。
Pass the names as parameters to the script, rather than substituting them into the script (as xargs
would be doing if you fixed its usage by adding -I
or -J
parameters: 将名称作为参数传递给脚本,而不是将其替换为脚本(如
xargs
可以通过添加-I
或-J
参数来固定其用法,这样做:
# best-practice approach: run a completely fixed shell script, passing arguments on
# its command line.
xargs -P 8 -n 1 \
sh -c 'for x; do csh "${x}.csh" >"${x}.log" 2>&1; done' _
You'll note that there's a sh -c
instance invoked: This is needed because xargs
itself doesn't understand shell operations such as redirections; 您会注意到有一个
sh -c
实例被调用:这是必需的,因为xargs
本身不了解诸如重定向之类的shell操作。 if you want a redirection to be performed, you need a shell to do it. 如果要执行重定向,则需要一个外壳来执行。
Now, let's go a little more into why your original code behaved as it did: 现在,让我们进一步探讨为什么原始代码会如此表现:
xargs -P 8 % csh '%'.csh >& '%'.log
... first performs the redirection to %.log
, then runs the command ... 首先执行重定向到
%.log
, 然后运行命令
xargs -P 8 % csh '%'.csh
There's no opportunity for xargs
to replace the %.log
string, because that redirection was performed by the enclosing shell before the xargs
command was run at all. xargs
没有机会替换%.log
字符串,因为重定向是在运行xargs
命令之前由封闭的外壳程序执行的。
Using GNU Parallel it looks like this: 使用GNU Parallel看起来像这样:
cat temp | parallel -P 8 'csh {}.csh >& {}.log'
If you have 8 cores you can even do: 如果您有8个核心,则可以执行以下操作:
cat temp | parallel 'csh {}.csh >& {}.log'
GNU Parallel quotes {} so that malicious input is will not be executed. GNU并行引号{}使得不会执行恶意输入。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.