MPI_send MPI_recv failed while increase the arry size

Question

I am trying to write a 3D parallel computing Poisson solver using OpenMPI ver 1.6.4.

The following parts are my code for parallel computing using blocking send receive.

The following variable is declared in another file.

int px = lx*meshx; //which is meshing point in x axis.
int py = ly*meshy;
int pz = lz*meshz;
int L = px * py * pz

The following code works well while

lx=ly=lz=10;

meshx=meshy=2, meshz=any int number.

The send recv parts failed while meshx and meshy are larger than 4.

The program hanging there waiting for sending or receiving data.

But it works if I only send data from one processor to another, not exchange the data. (ie : send from rank 0 to 1, but dont send from 1 to 0)

I can't understand how this code works while meshx and meshy is small but failed while mesh number xy in large.

Does blocking send receive process will interrupt itself or I confuse the processor in my code?Does it matter with my array size?

#include "MPI-practice.h"

# include <iostream>
# include <math.h>
# include <string.h>
# include <time.h>
# include <sstream>
# include <string>

# include "mpi.h"


using namespace std;


extern int px,py,pz;
extern int L;
extern double simTOL_phi;
extern vector<double> phi;

int main(int argc, char *argv[]){

    int   numtasks, taskid, offset_A, offset_B, DD_loop,s,e;
    double errPhi(0),errPhi_sum(0);

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numtasks);
    MPI_Comm_rank(MPI_COMM_WORLD, &taskid);
    MPI_Status status;

    if((pz-1)%numtasks!=0){
        //cerr << "can not properly divide meshing points."<<endl;
        exit(0);
    }

    offset_A=(pz-1)/numtasks*px*py;
    offset_B=((pz-1)/numtasks+1)*px*py;

    s=offset_A*taskid;
    e=offset_A*taskid+offset_B;


    int pz_offset_A=(pz-1)/numtasks;
    int pz_offset_B=(pz-1)/numtasks+1;

    stringstream name1;
    string name2;

    Setup_structure();
    Initialize();
    Build_structure();

    if (taskid==0){
        //master processor

        ofstream  output;
        output.open("time", fstream::out | fstream::app);
        output.precision(6);

        clock_t start,end;

        start=clock();

        do{
            errPhi_sum=0;

            errPhi=Poisson_inner(taskid,numtasks,pz_offset_A,pz_offset_B);
            //Right exchange

            MPI_Send(&phi[e-px*py], px*py, MPI_DOUBLE, taskid+1, 1, MPI_COMM_WORLD);
            MPI_Recv(&phi[e], px*py, MPI_DOUBLE, taskid+1, 1, MPI_COMM_WORLD, &status);

            MPI_Allreduce ( &errPhi, &errPhi_sum, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD );

        }while(errPhi_sum>simTOL_phi);

        end=clock();
        output << "task "<< 0 <<" = "<< (end-start)/CLOCKS_PER_SEC <<endl<<endl;

        Print_to_file("0.txt");

        //recv from slave
        for (int i=1;i<numtasks;i++){
            MPI_Recv(&phi[offset_A*i], offset_B, MPI_DOUBLE, i, 1, MPI_COMM_WORLD, &status);
        }

        Print_to_file("sum.txt");

    }
    else{
        //slave processor

        do{

            errPhi=Poisson_inner(taskid,numtasks,pz_offset_A,pz_offset_B);

            //Left exchange
            MPI_Send(&phi[s+px*py], px*py, MPI_DOUBLE, taskid-1, 1, MPI_COMM_WORLD);
            MPI_Recv(&phi[s], px*py, MPI_DOUBLE, taskid-1, 1, MPI_COMM_WORLD, &status);



            //Right exchange
            if(taskid!=numtasks-1){
                MPI_Send(&phi[e-px*py], px*py, MPI_DOUBLE, taskid+1, 1, MPI_COMM_WORLD);
                MPI_Recv(&phi[e], px*py, MPI_DOUBLE, taskid+1, 1, MPI_COMM_WORLD, &status);
            }

            MPI_Allreduce ( &errPhi, &errPhi_sum, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD );

        }while(errPhi_sum>simTOL_phi);

        //send back master
        MPI_Send(&phi[s], offset_B, MPI_DOUBLE, 0, 1, MPI_COMM_WORLD);

        name1<<taskid<<".txt";
        name2=name1.str();
        Print_to_file(name2.c_str());


    }

    MPI_Finalize();
}

Answer 1

Replace all coupled MPI_Send/MPI_Recv calls with a calls to MPI_Sendrecv . For example, this

MPI_Send(&phi[e-px*py], px*py, MPI_DOUBLE, taskid+1, 1, MPI_COMM_WORLD);
MPI_Recv(&phi[e], px*py, MPI_DOUBLE, taskid+1, 1, MPI_COMM_WORLD, &status);

becomes

MPI_Sendrecv(&phi[e-px*py], px*py, MPI_DOUBLE, taskid+1, 1,
             &phi[e], px*px, MPI_DOUBLE, taskid+1, 1,
             MPI_COMM_WORLD, &status);

MPI_Sendrecv uses non-blocking operations internally and thus it does not deadlock, even if two ranks are sending to each other at the same time. The only requirement (as usual) is that each send is matched by a receive.

Answer 2

The problem is in your inner most loop. Both tasks do a blocking send at the same time, which then hangs. It doesn't hang with smaller data sets, as the MPI library has a big enough buffer to hold the data. But once you increase that beyond the buffer size, the send blocks both processes. Since neither process are trying to receive, neither buffer can empty and the program deadlocks.

To fix it, have the slave first receive from the master, then send data back. If your send/receive don't conflict, you can switch the order of the functions. Otherwise you need to create a temporary buffer to hold it.