MPI_Comm_spawn and MPI_Reduce

Question

I have two programs. The "master" which spawns "workers" which perform some calculations and I want the master to get the results from the workers and store the sum. I am trying to use MPI_Reduce to collect the results from the workers, and the workers use MPI_Reduce to send to the masters MPI_Comm. I am not sure if that is correct. Here are my programs:

Master:

#include <mpi.h>
#include <iostream>
using namespace std;

int main(int argc, char *argv[]) { 
    int world_size, universe_size, *universe_sizep, flag; 

    int rc, send, recv;

    // intercommunicator
    MPI_Comm everyone;

    MPI_Init(&argc, &argv); 
    MPI_Comm_size(MPI_COMM_WORLD, &world_size); 

    if (world_size != 1) {
        cout << "Top heavy with management" << endl;
    } 

    MPI_Attr_get(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, &universe_sizep, &flag);  
    if (!flag) { 
        cout << "This MPI does not support UNIVERSE_SIZE. How many processes total?";
        cout << "Enter the universe size: ";
        cin >> universe_size; 
    } else {
        universe_size = *universe_sizep;
    }
    if (universe_size == 1) {
        cout << "No room to start workers" << endl;
    }

    MPI_Comm_spawn("so_worker", MPI_ARGV_NULL, universe_size-1,  
             MPI_INFO_NULL, 0, MPI_COMM_SELF, &everyone,  
             MPI_ERRCODES_IGNORE);

    send = 0;

    rc = MPI_Reduce(&send, &recv, 1, MPI_INT, MPI_SUM, 0, everyone);

    // store result of recv ...
    // other calculations here
    cout << "From spawned workers recv: " << recv << endl;

    MPI_Finalize(); 
    return 0; 
}

Worker:

#include <mpi.h>
#include <iostream>
using namespace std;

int main(int argc, char *argv[]) { 

    int rc, send,recv;


    int parent_size, parent_id, my_id, numprocs; 
    // parent intercomm
    MPI_Comm parent; 
    MPI_Init(&argc, &argv); 

    MPI_Comm_get_parent(&parent); 
    if (parent == MPI_COMM_NULL) {
        cout << "No parent!" << endl;
    }
    MPI_Comm_remote_size(parent, &parent_size); 
    MPI_Comm_rank(parent, &parent_id) ; 
    //cout << "Parent is of size: " << size << endl;
    if (parent_size != 1) {
        cout << "Something's wrong with the parent" << endl;
    }

    MPI_Comm_rank(MPI_COMM_WORLD, &my_id) ;     
    MPI_Comm_size(MPI_COMM_WORLD, &numprocs) ;  

    cout << "I'm child process rank "<< my_id << " and we are " << numprocs << endl;
    cout << "The parent process rank "<< parent_id << " and we are " << parent_size << endl;

    // get value of send
    send = 7; // just an example
    recv = 0;

    rc = MPI_Reduce(&send, &recv, 1, MPI_INT, MPI_SUM, parent_id, parent);
    if (rc != MPI_SUCCESS)
        cout << my_id << " failure on mpi_reduce in WORKER" << endl;

    MPI_Finalize(); 
    return 0; 
} 

I compiled both and execute like this (mpic++ for osx):

mpic++ so_worker.cpp -o so_worker
mpic++ so_master.cpp -o so_master
mpirun -n 1 so_master

Is this the correct way to run a master that spawns the workers?

In the master I always get 0 back from the MPI_Reduce. Can I use MPI_reduce from intercommunicators or should I use MPI_Send from workers and MPI_Recv from master? I'm really not sure why it's not working.

Any help would be appreciated. Thanks!

Answer 1

MPI_Comm_get_parent returns the parent intercommunicator that encompasses the original process and all the spawned ones. In this case calling MPI_Comm_rank(parent, &parent_id) does not return the rank of the parent but rather the rank of the current process in the local group of the intercommunicator:

I'm child process rank 0 and we are 3
The parent process **rank 0** and we are 1
I'm child process rank 1 and we are 3
The parent process **rank 1** and we are 1
I'm child process rank 2 and we are 3
The parent process **rank 2** and we are 1

(observe how the highlighted values differ - one would expect that the rank of the parent process should be the same, shouldn't it?)

That's why the MPI_Reduce() call would not succeed as all worker processes specify different values for the root rank. Since originally there was one master process, its rank in remote group of parent would be 0 and hence all workers should specify 0 as the root to MPI_Reduce :

//
// Worker code
//
rc = MPI_Reduce(&send, &recv, 1, MPI_INT, MPI_SUM, 0, parent);

This is only half of the problem. The other half is that rooted collective operations (eg MPI_REDUCE ) operate a bit different with intercommunicators. One first has to decide which of the two groups would host the root. Once the root group is identified, the root process has to pass MPI_ROOT as the value of root in MPI_REDUCE and all other processes in the root group must pass MPI_PROC_NULL . That is the processes in the receiving group do not take part in the rooted collective operation at all. Since the master code is written so that there could be only one process in the master's group, then it would suffice to change the call to MPI_Reduce in the master code to:

//
// Master code
//
rc = MPI_Reduce(&send, &recv, 1, MPI_INT, MPI_SUM, MPI_ROOT, everyone);

Note that the master also does not participate in the reduction operation itself, eg the value of sendbuf ( &send in this case) is irrelevant as the root would not be sending data to be reduced - it merely collects the result of the reduction performed over the values from the processes in the remote group.