按顺序运行作业而不是连续使用bash

Question

So I work a lot with Gaussian 09 (the computational chemistry software) on a supercomputer.

To submit a job I use the following command line

 g09sub input.com -n 2 -m 4gb -t 200:00:00

Where n is the number of processors used, m is the memory requested, and t is the time requested.

I was wondering if there was a way to write a script that will submit the first 10 .com files in the folder and then submit another .com file as each finishes.

I have a script that will submit all the .com files in a folder at once, but I have a limit to how many jobs I can queue on the supercomputer I use.

The current script looks like

 #!/bin/bash 
 #SBATCH --partition=shared
 for i in *.com
        do g09sub $i -n 2 -m 4gb -t 200:00:00
 done

So 1.com, 2.com, 3.com, etc would be submitted all at the same time.

What I want is to have 1.com, 2.com, 3.com, 4.com, 5.com, 6.com, 7.com, 8.com, 9.com, and 10.com all start at the same time and then as each of those finishes have another .com file start. So that no more than 10 jobs from any one folder will be running at the same time.

If it would be useful, each job creates a .log file when it is finished.

Though I am unsure if it is important, the supercomputer uses a PBS queuing system.

Answer 1

Try xargs or GNU parallel

xargs

ls *.com | xargs -I {} g09sub -P 10 {} -n 2 -m 4gb -t 200:00:00

Explanation:

• -I {} tell that {} will represent input file name
• -P 10 set max jobs at once

parallel

ls *.com | parallel -P 10 g09sub {} -n 2 -m 4gb -t 200:00:00 # GNU parallel supports -P too
ls *.com | parallel --jobs 10 g09sub {} -n 2 -m 4gb -t 200:00:00

Explanation:

• {} represent input file name
• --jobs 10 set max jobs at once

Answer 2

Not sure about the availability on your supercomputer, but the GNU bash manual offers a parallel example under 3.2.6 GNU Parallel , at the bottom.

There are ways to run commands in parallel that are not built into Bash. GNU Parallel is a tool to do just that.

...

Finally, Parallel can be used to run a sequence of shell commands in parallel, similar to 'cat file | bash'. It is not uncommon to take a list of filenames, create a series of shell commands to operate on them, and feed that list of commands to a shell. Parallel can speed this up. Assuming that file contains a list of shell commands, one per line,

parallel -j 10 < file

will evaluate the commands using the shell (since no explicit command is supplied as an argument), in blocks of ten shell jobs at a time.

---

Where that option was not available to me, using the jobs function worked rather crudely. eg:

for entry in *.com; do
   while [ $(jobs | wc -l) -gt 9 ]; do
     sleep 1    # this is in seconds; your sleep may support 'arbitrary floating point number'
   done
   g09sub ${entry} -n 2 -m 4gb -t 200:00:00 & 
done

$(jobs | wc -l) counts the number of jobs spawned in the background by ${cmd} &