如何使用MPI终止Signal

Question

This is my first program I wrote with MPI in c, The program is supposed to terminate within 15 secs, but it did not. it did not even go through the if(end_now == 1) statement. Does anyone know what had happened here? The code is simplified as below:

int end_now = 0;


void sig_handler(int signo)
{
    if (signo == SIGUSR1) {
        end_now = 1;
        printf ( "  %8d  %8d\n", current_number, current_total);
    }
}

int main ( int argc, char **argv ){
   int id;
   int count;

   MPI_Init (&argc, &argv);
   MPI_Comm_size (MPI_COMM_WORLD, &count);
   MPI_Comm_rank (MPI_COMM_WORLD, &id);

   signal(SIGUSR1, sig_handler);


   while (1){
       //MPI_Allreduce is called here to sum up the subtotal calculated by child processes
        if (end_now == 1){
            printf("here\n");  //this "here is never printed out"
            break;
        }
    }
    MPI_Finalize ();
    return 0;
}

I am using timeout --signal=USR1 15 mpirun.openmpi -np 2 ./a.out to execute the code on my Mac. Thanks to whoever may help.

Answer 1

You're sending a signal to mpirun , not your executable. Since mpirun itself doesn't have a SIGUSR1 handler, nothing happens.

BTW, using signals with MPI programs is not something you want to do. MPI programs rely on multiple invocations running in lockstep, which doesn't match the asynchronous and per-process nature of signals.

Answer 2

Sneftel is right. Gilles Gouaillardet is also VERY right. I want to add some other info.

Even if you send the signal to actual program and not to "mpirun", then you possibly send it to ONE only of your processes and not to all of them.

Yes, signals are not right thing to do in MPI programs. But even if you want to use them, you should first debug whatever processes get them and who of them get them.

Insert "printf" directly into signal handler. Print something like "MPI process number %d got the signal" and insert MPI_COMM_RANK into this printf. (UPD 2018-04-27 7:31 MSK: sorry, I didn't noticed you already have such printf in your code.) (Note: I think "printf" in MPI programs allowed in first process only, and using "printf" in other processes is probably bad idea, but for debugging purposes will go. Also, I think "printf" directly from signal handler is bad idea, but, again, for debugging purposes will go.)

You will determine if your processes get the signal and which of them.

If you don't satisfied with results, then try different programs instead of gtimeout. For example, "timeout" from GNU Coreutils. (Well, this is Mac, I'm not sure, GNU Coreutils is available form Mac, but I still think you can find SOME "timeout".)

Then: you didn't describe your setup in question. Does your MPI programs run on different hosts or on one? Does MPI "programs" really implemented as separate programs or as threads? Which MPI implementation you use and which version? If you don't know how MPI starts your processes, at least say us, how you installed your MPI implementation and how you configured it.

Or even you can do without any "timeout" or "gtimeout" at all. Just type this in one console:

sh -c 'echo $$ > ~/pid-of-mpirun; exec ~/opt/usr/local/bin/mpirun -np 2 ./a.out'

This will run "mpirun" while storing its PID into ~/pid-of-mpirun. And run in parallel in another terminal (of course, you don't need to run this command exactly in the same moment):

sleep 15; kill -USR1 $(cat ~/pid-of-mpirun)

This will want 15 secs and send USR1 to process which PID is in ~/pid-of-mpirun .

But all this will probably send USR1 to "mpirun" and not to actual processes (I am not sure, test this!). How to send to actual processes? Well, you can read manual page for "kill" and try to understand how to send a signal to whole process group and not to just one process.

Also, you can write your PID into some file directly inside your C program.

Example:

#include <stdio.h>
#include <unistd.h> // Mac is one of UNIX systems, so we have unistd.h
// ...
FILE *fout = fopen("~/my-pid", "w"); fprintf(fout, "%d\n", getpid); fclose(fout);

Of course, you should somehow make sure you create different files in different processes. For example, generate file names from MPI_COMM_RANK.

Answer 3

还应将end_now声明为volatile否则编译器可能会优化将永远运行的主循环。

Answer 4

我建议不要在信号处理程序中使用printf，因为printf是不可重入的函数，在某些平台上这可能导致程序崩溃。