了解2D MPI网格中的过程的尺寸，坐标和顺序

Question

I have 3 questions, all related to MPI (in C). I think the first 2 have the same answer, but I'm not positive.

---

Question 1

When using MPI_Dims_create to create a 2D grid, which dimension of the returned dimensions is X and which is Y?

For example:

int numProcs = -1, myProcID = -1;
MPI_Comm_rank(MPI_COMM_WORLD, &myProcID);
MPI_Comm_size(MPI_COMM_WORLD, &numProcs);

int size[2];
size[0] = size[1] = 0;
MPI_Dims_create(numProcs, 2, size);
int numProcs_y = -1, numProcs_x = 1;

Should it be:

numProcs_y = size[0];
numProcs_x = size[1];

Or this:

numProcs_x = size[0];
numProcs_y = size[1];

I tried running this with numbers of processes that seemed like they would give me the answer (eg 6). With 6 processes, MPI_Dims_create should either create a grid with 3 rows and 2 columns or 2 rows and 3 columns (unless I've misunderstood the documentation). For process 3 (of 6), I found that size[0] = 1 and size[1] = 0 (I believe this corresponds to x = 0, y = 1). This seems to indicate to me that MPI_Dims_create is creating a grid with 3 rows and 2 columns (because it was 2 rows and 3 columns, process 2 (of 6) should have x = 2, y = 0). Any confirmation someone could provide on this would be greatly appreciated.

---

Question 2

When using MPI_Cart_coords on a 2D Cartesian grid, which of the returned dimensions is X and which is Y?

For example:

int periodic[2];
periodic[0] = periodic[1] = 0; // no wrap around
MPI_Comm cart_comm;
int coords[2];

// using size from question 1
MPI_Cart_create(MPI_COMM_WORLD, 2, size, periodic, 1, &cart_comm);
MPI_Cart_coords(cart_comm, myProcID, 2, coords);

Similar to question 1, my question is, should it be like this

myProcID_y = coords[0];
myProcID_x = coords[1];

or like this

myProcID_x = coords[0];
myProcID_y = coords[1];

I've been searching through the documentation and previous questions on here, but I can't seem to find a direct answer to this question.

The documentation seems to indicate that the first of the two approaches is the correct one, but it doesn't definitively state it.

---

Question 3

The underlying question behind the first 2 questions is that I'm trying to split the 2D grid into rows and columns. However, when I use MPI_Comm_rank to check the row and column IDs of each process after doing so, I'm getting the processes ordered in an order that doesn't match with what I think the answers to the above questions are.

Based on above, I expect the processes to be ordered like this:

P0 P1
P2 P3
P4 P5

However, using this code (which comes after the above code in my program, so all the above code is accessible from it: I'm trying to separate out my code to make it easier to isolate my questions):

MPI_Comm row_comm, col_comm;
int row_id = -1, col_id = -1;
// ** NOTE: My use of myProcID_y and myProcID_x here are based 
// on my understanding of the previous 2 questions ... if my 
// understanding to one/both of those is wrong, then obviously the assignments 
// here are wrong too.
// create row and column communicators based on my location in grid
MPI_Comm_split(cart_comm, myProcID_y, myProcID_x, &row_comm);
MPI_Comm_split(cart_comm, myProcID_x, myProcID_y, &col_comm);

// get row and column ID for each process
MPI_Comm_rank(row_comm, &row_id);
MPI_Comm_rank(col_comm, &col_id);
printf("Process: %d\trowID: %d\tcolID: %d\n", myProcID, row_id, col_id);

I'm seeing the following printed:

Process: 0 rowID: 0        colID: 0
Process: 1 rowID: 1        colID: 0
Process: 2 rowID: 0        colID: 1
Process: 3 rowID: 1        colID: 1
Process: 4 rowID: 0        colID: 2
Process: 5 rowID: 1        colID: 2

which seems to correspond to the following order of processes:

P0 P2 P4
P1 P3 P5

which is the opposite matrix dimensions (2 rows, 3 columns) from what I was expecting from MPI_Dims_create .

Assuming my understanding to the first 2 questions is correct (ie that Y is the first dimension returned by those functions), why are the processes (seemingly) being ordered in a different order at this step?

Answer 1

Q1 and Q2. There are no such things as dimension X and dimension Y in MPI - these are labels that you give to the abstract dimensions used in the standard. MPI works with numbered dimensions and follows the C row-major numbering of the ranks, ie in a 2x3 Cartesian topology (0,0) maps to rank 0 , (0,1) maps to rank 1 , (0,2) maps to rank 2 , (1,0) maps to rank 3 , and so on.

Note that dimension 0 corresponds to the rightmost element in the coordinate tuple. This is in reverse to the numbering of elements in C arrays and often a source of confusion. To create a 2x3 Cartesian topology the size array would have to be initialised as:

int size[2] = { 3, 2 };

It is up to you to map the abstract numbered dimensions to your problem. You may chose dimension 0 to be X, or you may chose dimension 1 - it doesn't matter.

As for MPI_DIMS_CREATE , the standard says:

The dimensions are set to be as close to each other as possible, using an appropriate divisibility algorithm.

For dims[i] set by the call, dims[i] will be ordered in non-increasing order.

This operation simply returns an array of elements dims[i] that have the following properties (unless the size of one or more dimensions is fixed by setting a non-zero value(s) in dims[] before calling MPI_DIMS_CREATE ):

• dims[0] >= dims[1] >= dims[2] >= ...
• dims[0] * dims[1] * dims[2] == nprocs , where nprocs is the number of processes, specified to MPI_DIMS_CREATE .

This means that as MPI_DIMS_CREATE decomposes the set of nprocs into an multidimensional grid, it would assign the biggest multiplier to the size of dimension 0 , the next one to the size of dimension 1 , and so on. As 6 factors like 2*3 , then MPI_DIMS_CREATE would return { 3, 2 } . If you call MPI_CART_CREATE directly with the result from MPI_DIMS_CREATE , it would create a 2x3 topology with coordinates and ranks:

(0,0)=0 (0,1)=1 (0,2)=2
(1,0)=3 (1,1)=4 (1,2)=5

Q3. MPI provides a special routine to partition Cartesian topologies - MPI_CART_SUB . It takes an array of logical flags (integers in C), named remain_dims in the standard. Each non-zero remain_dims[i] means that that dimension i should be retained in the resulting partitioning while separate subcommunicators would be created for any possible combination of the non-retained dimensions. For example, given the 2x3 topology:

• remain_dims[] = { 1, 0 } would retain dimension 0 and result in 2 non-overlapping 1-d communicators with 3 processes each;
• remain_dims[] = { 0, 1 } would retain dimension 1 result in 3 non-overlapping 1-d communicators with 2 processes each;
• remain_dims[] = { 0, 0 } would not retain any of the two dimensions and will result in 6 non-overlapping zero-dimensional communicators with a single process in each.

Which partitioning you would call row-wise and which one you would call a column-wise partitioning is up to you and your labelling of the Cartesian dimensions.

---

One thing to note is that one often runs MPI codes on systems that consist of networked SMP or NUMA multicore nodes. In this case a suitable 2D grid would be nodes x cores . If the number of cores is known, then one can easily fix it in the call to MPI_DIMS_CREATE :

int size[2] = { ncores, 0 };

MPI_Dims_create(numProcs, 2, size);

This is a more convenient than dividing numProcs by ncores and checking for divisibility as MPI_DIMS_CREATE would signal an error if ncores does not divide numProcs . Then a partitioning that keeps dimension 0 , ie one with

int remain_dims[2] = { 1, 0 };

would create subcommunicators that contain processes on the same node, while

int remain_dims[2] = { 0, 1 };

would create subcommunicators that contain no two processes from the same node.

---

Note that in your code you have specified a value of 1 (true) to the reorder parameter in MPI_CART_CREATE . This might lead to processes having different ranks in MPI_COMM_WORLD and in the Cartesian communicator. Hence there is no guarantee that the following lines of code do what you expect them to:

MPI_Cart_coords(cart_comm, myProcID, 2, coords);
                           ^^^^^^^^
...
printf("Process: %d\trowID: %d\tcolID: %d\n", myProcID, row_id, col_id);
                                              ^^^^^^^^

myProcID was obtained from MPI_COMM_WORLD and might actually differ from the new rank in cart_comm , hence it should not be used to obtain the process coordinates and to perform the splits.