Is there a way to speed up the setting of multiple parameters in different bash files?

Question

I'm using the SciNet computers for some simulation work, and part of this requires me to simulate data and test models using many different parameter combinations that have to be set in each bash file individually before running them through the terminal. These are just.sh files that run aR script with certain options set, and I edit all of them using a simple text editor on Windows.

This can take a lot of time, however, if I have to specify over 100 combinations by hand in a text file. I'm wondering, is there is any way to speed up this process? Possibly a script or program that I can run to copy from one base file and make according changes to the parameters. Or if there was a way I could create a dataset and fill it with columns of parameter values that the bash file could then pull from that would be great. I haven't found anything in my search so far but I feel I may be looking in the wrong places.

Currently I can make some shortcuts by copying and pasting the files for simulations that have the most similarities between parameters, but this still requires me to go into each file manually to change the parameters that do not match.

The main body of the parameters for one setting looks as follows:

--studies 10 --rate 0.003 --alpha 3 --beta 3 --reps 10 --mc 500 --job ${SLURM_ARRAY_TASK_ID} --out /Results

Answer 1

The way you described the problem is confusing. Hence, the number of comments.

The below script is what I visualized to be the issue you are grappling with. Namely,

• codifying scenario parameters, and
• looping using those parameters.

The script:

#!/bin/bash

DBG=0

COMMAND_ROOT="something_mix_of_command_and_shared_parameters"

###
### Defining Scenario Parameters
###
rates=( 0.001 0.002 0.003 0.004 0.005 )
alphas=( 1 2 3 )
betas=( 1 2 3 )
reps=( 5 10 20 50 100 )
mcs=( 400 450 500 550 600 )

###
### Looping on parameters for "test" plan
###
for rate in ${rates[@]}
do
    test ${DBG} -eq 1 && echo -e "\t [rate= ${rate}] ..."
    for alpha in ${alphas[@]}
    do
        test ${DBG} -eq 1 && echo -e "\t\t [alpha= ${alpha}] ..."
        for beta in ${betas[@]}
        do
            test ${DBG} -eq 1 && echo -e "\t\t\t [beta= ${beta}] ..."
            for rep in ${reps[@]}
            do
                test ${DBG} -eq 1 && echo -e "\t\t\t\t [rep= ${rep}] ..."
                for mc in ${mcs[@]}
                do
                    test ${DBG} -eq 1 && echo -e "\t\t\t\t\t [mc= ${mc}] ..."
                    SLURM_ARRAY_TASK_ID="${some_unique_identifier}"

                    COMMAND_PARAMETERS="\
                    --studies 10 \
                    --rate ${rate} \
                    --alpha ${alpha} \
                    --beta ${beta} \
                    --reps ${rep} \
                    --mc ${mc} \
                    --job ${SLURM_ARRAY_TASK_ID} --out /Results"

                    echo -e " Running permutation [${rate}|${alpha}|${beta}|${rep}|${mc}] ..."

                    #${COMMAND_ROOT} ${COMMAND_PARAMETERS}
                done
                echo ""
            done
        done
    done
done
exit