Bash并行处理多个输入文件，一个输出文件

Question

I created a for loop that goes through multiple files and outputs the results into one file:

for x in /home/moleculo/x*; do ExtractOutCalls2.sh /home/Scripts/000 $x & done

So each of my input files starts with letter x, that's x* as input. Script takes each of those input files $x and outputs to file /home/Scripts/000

Now I have a question:

if this is done on a few thousand files, is it a good way to put like this?

also if I use multiple input files, but specify one output file, will this mean, that my output will won't be appended? If not, how to do it

Regards, Irek

Answer 1

Yes, your output file gets overwritten by each process. Make each script output to its own file, and once all the scripts are finished, concatenate the output:

i=0
for x in /home/moleculo/x* ; do
    ExtractOutCalls2.sh /home/Scripts/000 $x > OUT.$i &
    (( i++ ))
done
wait
cat OUT.* > OUT
rm OUT.*

You have to change the script to output to standard output instead of the file, or make it accept the name of the output file to be created.

Answer 2

Often you can use the file - to designate stdout:

for x in /home/moleculo/x*; do ExtractOutCalls2.sh - $x & done

To avoid mixing output use GNU Parallel:

parallel ExtractOutCalls2.sh - {} ::: /home/moleculo/x* > output