Shell script, parallel and multithreaded?

Question

I have a shell script which actually has file/folder locations and command to run a javascript. I am running the job on a linux cluster with 8 processors and 4 CPU cores per processor. I want to run this on multiple processors, at the same time each job accessing multiple threads to reduce the total runtime of the script. My question is: Is such a thing possible? If yes what would be the command or code snippet for this?

Answer 1

You may be able to use GNU parallel . Example:

parallel "zcat {} | bzip2 >{.}.bz2 && rm {}" ::: *.gz

"This will recompress all files in the current directory with names ending in .gz using bzip2, running one job per CPU (-j+0) in parallel."

---

If you don't have parallel, then you can roll your own coordinator in a couple of ways:

1. Use make -j
2. Use bash in-built backgrounding ( & )

You can use ssh for running commands on remote machines, and you can use nfs or its ilk for networked storage. However you do it, you will need to think about how to partition your jobs so that they can cooperate and, as needed, coalesce results.