MPI Programming in C - MPI_Send() and MPI_Recv() Address Trouble

Question

I'm currently working on a C program using MPI, and I've run into a roadblock regarding the MPI_Send() and MPI_Recv() functions, that I hope you all can help me out with. My goal is to send (with MPI_Send()), and receive (with MPI_Recv()), the address of "a[0][0]" (Defined Below), and then display the CONTENTS of that address after I've received it from MPI_Recv(), in order to confirm my send and receive is working. I've outlined my problem below:

I have a 2-d array, "a", that works like this: a[0][0] Contains my target ADDRESS *a[0][0] Contains my target VALUE

ie printf("a[0][0] Value = %3.2f, a[0][0] Address = %p\\n", *a[0][0], a[0][0]);

So, I run my program and memory is allocated for a. Debug confirms that a[0][0] contains the address 0x83d6260, and the value stored at address 0x83d6260, is 0.58. In other words, "a[0][0] = 0x83d6260", and "*a[0][0] = 0.58".

So, I pass the address, "a[0][0]", as the first parameter of MPI_Send(): -> MPI_Send(a[0][0], 1, MPI_FLOAT, i, 0, MPI_COMM_WORLD); // I put 1 as the second parameter becasue I only want to receive this one address

MPI_Send() executes and returns 0, which is MPI_SUCCESS, which means that it succeeded, and my Debug confirms that "0x83d6260" is the address passed.

However, when I attempt to receive the address by using MPI_Recv(), I get Segmentation fault: MPI_Recv(a[0][0], 1, MPI_FLOAT, iNumProcs-1, 0, MPI_COMM_WORLD, &status);

The address 0x83d6260 was sent successfully using MPI_Send(), but I can't receive the same address with MPI_Recv(). My question is - Why does MPI_Recv() cause a segment fault? I want to simply print the value contained in a[0][0] immediately after the MPI_Recv() call, but the program crashes.

Answer 1

I think the problem is that you're trying to put the value into the array of pointers (which is probably causing the segfault). Try making a new buffer to receive the value:

MPI_Send(a[0][0], 1, MPI_FLOAT, i, 0, MPI_COMM_WORLD);

....

double buff;
MPI_Recv(&buff, 1, MPI_FLOAT, iNumProcs-1, 0, MPI_COMM_WORLD, &status);

If I remember correctly the MPI_Send/Recv will dereference the pointer giving you the value, not the address.

You also haven't given us enough information to tell if your source/destination values are correct.

Answer 2

MPI_Send(a[0][0], 1, MPI_FLOAT ...) will send memory with size sizeof(float) starting at a[0][0]

So basicaly the value sent is *(reinterpret_cast<float*>(a[0][0]))

Therefore if a[0][0] is 0x0x83d6260 and *a[0][0] is 0.58f then MPI_Recv(&buff, 1, MPI_FLOAT...) will set buffer (of type float, which need to be allocated) to 0.58

On important thing is that different MPI programm should NEVER share pointers (even if they run on the same node). They do not share virtual memory pagination and event if you where able to acces the adress from one on the rank, the other ones should give you a segfault if you try to access the same adress in their context

EDIT

This code works for me :

#include <stdio.h>
#include <stdlib.h>
#include "mpi.h"
int main(int argc, char* argv[])
{
    int size, rank;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    switch(rank)
    {
        case 0:
        {
            float*** a;
            a        = malloc(sizeof(float**));
            a[0]     = malloc(sizeof(float* ));
            a[0][0]  = malloc(sizeof(float  ));
            *a[0][0] = 0.58;
            MPI_Send(a[0][0], 1, MPI_FLOAT, 1, 0, MPI_COMM_WORLD);
            printf("rank 0 send done\n");
            free(a[0][0]);
            free(a[0]   );
            free(a      );
            break;
        }
        case 1:
        {
            float buffer;
            MPI_Recv(&buffer, 1, MPI_FLOAT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
            printf("rank 1 recv done : %f\n", buffer);
            break;
        }
    }
    MPI_Finalize();
    return 0;
}

results are :

mpicc mpi.c && mpirun ./a.out -n 2
> rank 0 send done
> rank 1 recv done : 0.580000