等待子进程完成，然后再开始新的子进程

Question

I have to process ten very big files. Each file takes about two days to process by my_profiler . I can parallelize the work so that my_profiler runs on each file separately, hence using all of my system's cores. My approach at parallelizing the work is to run three processes in three different terminals at the time. I can't process more than four files at once, or my system starts getting unresponsive (hangs up).

My goal is to write a shell script which processes the ten files in batches of size three. Once processing of one file finishes, the terminal should be closed and processing of a new file should start in another terminal. As a terminal I want to use gnome-terminal .

Currently I am stuck with the following script, which runs all processes in parallel:

for j in $jobs
do
    gnome-terminal -- bash -c "my_profiler $j"
done

How I can wait until a shell script running in a instance of gnome-terminal finishes?

My first thought was that I might need to send a signal form the old terminals once their job is finished.

Answer 1

I am not quite sure why you have to start a new gnome-terminal for each job. But you could use xargs in combination with -P ^[1] . Running three my_profiler in parallel at the same time:

echo "${jobs}" | xargs -P3 -I{} gnome-terminal --wait -e 'bash -c "my_profiler {}"'

Important here is to start gnome-terminal with --wait otherwise the terminal demonizes itself which will have the effect that xargs starts the next process. --wait was introduced with gnome-terminal 3.27.1 .

The -I{} option to xargs defines a placeholder ( {} ) which xargs will replace with a filename before running the command ^[2] . In the example above, xargs scans the command string ( gnome-terminal --wait -e 'bash -c "my_profiler {}"' ) for {} and replaces the found instances with the first file coming from stdin ( echo "${jobs}" | ... ). The resulting string it then executes. xargs will do this three times ( -P3 ), before it starts waiting for at least one process to finish. If this happens, xargs will start the next process.

---

[1]: from man xargs

-P max-procs , --max-procs=max-procs

Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n option or the -L option with -P ; otherwise chances are that only one exec will be done. While xargs is running, you can send its process a SIGUSR1 signal to increase the number of commands to run simultaneously, or a SIGUSR2 to decrease the number. You cannot increase it above an implementation-defined limit (which is shown with --show-limits ). You cannot decrease it below 1. xargs never terminates its commands; when asked to decrease, it merely waits for more than one existing command to terminate before starting another.

Please note that it is up to the called processes to properly manage parallel access to shared resources. For example, if more than one of them tries to print to stdout, the ouptut will be produced in an indeterminate order (and very likely mixed up) unless the processes collaborate in some way to prevent this. Using some kind of locking scheme is one way to prevent such problems. In general, using a locking scheme will help ensure correct output but reduce performance. If you don't want to tolerate the performance difference, simply arrange for each process to produce a separate output file (or otherwise use separate resources).

[2]: from man xargs

-I replace-str

Replace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks do not terminate input items; instead the separator is the newline character. Implies -x and -L 1 .

Answer 2

If I understand this right...

I think you could use wait $job in order for the job to complete.

Here's an example. Following script will start max. 3 jobs in parallel, in background. Once one of these 3 jobs will end, it will start another one.

#!/bin/bash

THREADS='3';
FILES=$(find source_dir_path -type f -name "your files*")

for file in ${FILES}
do
 NUMPROC=$(ps -ef |grep -i [y]our_process_name| wc -l |tr -d ' ')
 while (( $NUMPROC >= 3))
 do
  sleep 60
  NUMPROC=$(ps -ef |grep -i [y]our_process_name| wc -l |tr -d ' ')
 done
 echo "Starting: " $file;
 #your file processing command below, I assume this would be:
 my_profiler $file &
done

for job in `jobs -p`
do
 wait $job
done

Answer 3

Each file takes about 2 days to process

Running them in a graphical window is the most expensive operation there is. Flushing the terminal window can be expensive, if your process outputs lot of stdout (like cp -vr /bigfolder /anotherfolder ) you will see performance difference. Also, running an X application with background job makes it dependent on X server - if your X server crashes, you loose your work. That all is unrelated work to what you are trying to do.

For single run workloads (run&forget), I would go with xargs -Pjobs . I would add some ionice nice to make the system usable white the process is running. The processes stdout output could be discarded, interleaved with some prefix added ex. with | sed 's/^/'"${job}: "'/' | sed 's/^/'"${job}: "'/' , saved to a file. Or better, | logger | logger redirected to system logger.

If it would be a one-time job, I would open a tmux or screen session, type:

printf "%s\n" $jobs | ionice nice xargs -t -P$(nproc) sh -c 'my_profiler "$1"' --

and discard the tmux or screen session for later use. Set an alarm on my phone in 3 days and check it in 3 days later.

The ionice nice will make your system somehow usable while the processes are processing. The -P$(nproc) will limit the processes to the number of cores. If the my_profiler is highly I/O dependent and you don't care about the system performance while running the jobs, It's sometimes advisable to run more jobs then cores, as they will block on I/O anyway.

You could add | logger -p local0.info --id=$$ | logger -p local0.info --id=$$ to the end after xargs or inside child sh shell inside xargs , so that it will redirect the output to system log using local0.info priority and id of the PID of the current shell.

A way better in my opinion option is to create a systemd service file. Create such my_profiles@.service file:

[Unit]
Description=Run my_profiler for %i
[Service]
# full path to my_profiler
ExecStart=/usr/bin/my_profiler %i
CPUSchedulingPolicy=batch
Nice=19
IOSchedulingClass=best-effort

Add the service to search path with systemd link my_profiler@.service or create that as a dropin service file inside /var/run/systemd/system . Then start running it with printf "%s\\n" $jobs | xargs -I{} -t systemctl start ./my_profiler@{}.service printf "%s\\n" $jobs | xargs -I{} -t systemctl start ./my_profiler@{}.service .

That way I could get all the logs I need from journalctl -u my_profiler@job.service and logs will never fill 100% of my disc space, because journalctl checks that. Errors will be easily reported and inspected with systemd list-failed or systemd status my_profiler@job.service .

Answer 4

Yet another approach, because counting the processes based on a substring in the process table might be problematic. Especially if you start subprocesses in your script the count can be unreliable. You also wrote, that the processes run for 2 days, so you might have the problem at times, that you need to restart from the prefious point.

You could do that in just a bit more complicated way. You need one script that starts your processes and watches them if they still look healty (the process did not crash --> otherwise it restarts them). This requires an init script, the script which fills the process queue and a small modification of your profiler script.

Script 1: initialize the process

Create a job directory with one file per job in order to track the progress automatically. If all jobs can be processed without problems, it will be deleted automatically later.

#!/bin/bash
tmpdir=/tmp/
jobdir=${tmpdir}/jobs
num_jobs=3
mkdir -p ${jobdir}

i=1
for file in $jobs ; do
    ((i++))
    echo "${file}" > ${jobdir}/${i}.open
done

Script 2: starting the actual processes

#!/bin/bash
jobdir=${tmpdir}/jobs
num_jobs=3

function fill_process_queue() {
    # arg1: num_jobs
    # arg2: jobdir
    # arg3...: open jobs
    num_jobs=$1
    jobdir=$2
    shift 2
    while [[ $(ls ${jobdir}/*.running.* | wc -l) -lt ${num_jobs} -a $# -gt 0 ]] ; do
        job_file=$1
        shift 1
        gnome-terminal -- bash -c "my_profiler $(cat ${jobdir}/${job_file}) ${jobdir}/${job_file}"
        # now give the called job some time to
        # mark it's territory (rename the job file)
        sleep 5s
    done
}

while [[ -z $(ls ${jobdir}) ]] ; do
    # still files present, so first check if
    # all started processes are still running
    for started_job in $(ls ${jobdir}/*.running.* 2>/dev/null) ; do
        # check if the running processes are still alive
        pid= "{started_job//[0-9]\.running\.}"
        jobid= "{started_job//\.running\.[0-9]*}"
        if ! kill -0 ${pid} 2> /dev/null ; then
            # process is not running anymore
            # don't worry kill -0 doesn't harm your
            # process
            mv ${jobdir}/${started_job} ${jobdir}/${jobid}
        fi
    done
    fill_process_queue ${num_jobs} ${jobdir} ${jobdir}/*.open
    sleep 30s
done
# if the directory is empty, it will be removed automatically by rmdir, if non-empty, it remains
rmdir ${jobdir}

Changes of the profiler script

The profiler script needs to rename the job file, so it includes the pid of the profiler script at the start of the script and needs to delete the file, once it is sucessfully finished. The file name is passed as extra argument after the job argument (so it should be argument 2). These changes look like:

# at the very beginning of your script
process_file=${2//\.open/}.running.$$
mv $2 ${process_file}

# at the very end of your script, if everything went fine
rm ${process_file}