用于在MPI中的3D过程分解中交换2D晕圈的子阵列数据类型的数量

Question

Assume a global cube of dimensions GX*GY*GZ which is decomposed using 3D Cartesian Topology into 3D cubes of sizes PX*PY*PZ on each process. Adding Halos for exchange of data this becomes (PX+2)*(PY+2)*(PZ+2) . Assuming we use the Subarray data type for 2D halo exchange - do we need to define 12 subarray types ?

My reasoning is this: For YZ plane we create one Subarray type for sending and one subarray type for receiving as the starting coordinates are to be specified WITHIN the Subarray data type itself. But there are 2 YZ planes, which results in 4 Subarray datatypes. Though the global and local data sizes remain the same but due to the starting indexes - we need to define 4 distinct Subarray types. Isn't it better to send four of these planes using a Vector data type and the remaining two using a Subarray data type ?

Answer 1

You have three pattens of data access here - sending/receiving an X-face of the subdomain, a Y-face, and a Z-face - so you need three different ways of describing those patterns. Which and how many types you use to describe that is largely dependant on what you find the clearest way of expressing and using those patterns.

Let's say you have, locally, PX=8, PY=5, PZ=7, so that including the halo, the local subdomains are 10x7x9. This is in C, so we'll assume the data is stored in some contiguous array arr[ix][iy][iz] , so that going values (ix,iy,1) and (ix,iy,2) are contiguous (offset by one item size - say 8 bytes for doubles), values (ix,1,iz) and (ix,2,iz) are offset by (PZ+2) [that is, 9] values, and (1,iy,iz) and (2,iy,iz) are offset by (PY+2)*(PZ+2) [ = 7*9 = 63 ] values.

So let's see how this plays out, sketching out faces of the grid with z/y being left/right and up/down, and x shown in neighbouring panels. For simplicity we'll include corner cells in what we send/receive.

The data you'd need to send an y-face to the upper neighbour looks like:

       x = 0          x = 1     ...      x = 9        Local Grid Size:
    +---------+    +---------+        +---------+     PX = 8
6   |         |    |         |        |         |     PY = 5
5   |@@@@@@@@@|    |@@@@@@@@@|        |@@@@@@@@@|     PZ = 7
4  ^|         |   ^|         |       ^|         |
3  ||         |   ||         |       ||         |
2  y|         |   y|         |       y|         |
1   |         |    |         |        |         |
0   |         |    |         |        |         |
    +---------+    +---------+        +---------+
     012345678      012345678   ...    012345678
        z->            z->                z->

That is, it would start at [0][PY][0] (eg, [0][5][0]) and extend to [PX+1][PY][PZ+1]. So you'd start at [0][PY][0]...[0][PY][PZ+1], which are PZ+2 contiguous values, and then go to [1][PY][0] - which is a jump of (PY+2)*(PZ+2) values from [0][PY][0], the start earlier, and take another PZ+2 contiguous values, and so on. You could express this simply as:

• MPI_Type_vector of count PX+2, blocklen (PZ+2), and stride of (PY+2)*(PZ+2), or
• MPI_Type_subarray, with a slice subsize of [PX+2,1,PZ+2], starting at [0,PY,0]

They are exactly equivalent, and there is no performance difference.

Now, let's consider receiving this data:

       x = 0          x = 1     ...      x = 9        Local Grid Size:
    +---------+    +---------+        +---------+     PX = 8
6   |         |    |         |        |         |     PY = 5
5   |         |    |         |        |         |     PZ = 7
4  ^|         |   ^|         |       ^|         |
3  ||         |   ||         |       ||         |
2  y|         |   y|         |       y|         |
1   |         |    |         |        |         |
0   |@@@@@@@@@|    |@@@@@@@@@|        |@@@@@@@@@|
    +---------+    +---------+        +---------+
     012345678      012345678   ...    012345678
        z->            z->                z->

Crucially, the data pattern needed is exactly the same: PZ+2 values, then skip (PY+2)*(PZ+2) values from the start of that last block, and another PZ+2 values. We could describe it as:

• MPI_Type_vector of count PX+2, blocklen (PZ+2), and stride of (PY+2)*(PZ+2), or
• MPI_Type_subarray, with a slice subsize of [PX+2,1,PZ+2], starting at [0,0,0]

The only difference is the starting position of the subarray for the subarray type. But this isn't as big a difference as it seems!

When you actually use the subarray type in a send or receive (say), you pass the routine a pointer to some data, and then give it a subarray type with some starting position and slice description. MPI then skips ahead to that starting position, and uses the data layout described by that slice.

So while it is perfectly fine to define and use four subarray types:

MPI_Type_create_subarray(ndims=3, sizes=[PX+2,PY+2,PZ+2], subsizes=[PX+2,1,PZ+2], 
                         starts=[0,0,0],... &recv_down_yface_t);
MPI_Type_create_subarray(...all the same...
                         starts=[0,1,0],... &send_down_yface_t);
MPI_Type_create_subarray(...all the same...
                         starts=[0,PY,0],... &send_up_yface_t);
MPI_Type_create_subarray(...all the same...
                         starts=[0,PY+1,0],... &recv_up_yface_t);

/* Send lower yface */
MPI_Send(&(arr[0][0][0]), 1, send_down_yface_t, ... );
/* Send upper yface */
MPI_Send(&(arr[0][0][0]), 1, send_up_yface_t, ... );
/* Receive lower face */
MPI_Recv(&(arr[0][0][0]), 1, recv_down_yface_t, ... );
/* Receive upper face */
MPI_Recv(&(arr[0][0][0]), 1, recv_up_yface_t, ... );

which declare four equivalent patterns with different starting points, you can also just define one, and use it pointing at different starting points for the data you need:

MPI_Type_create_subarray(ndims=3, sizes=[PX+2,PY+2,PZ+2], subsizes=[PX+2,1,PZ+2], 
                             starts=[0,0,0],... &yface_t);
/* ... */
/* Send lower yface */
MPI_Send(&(arr[0][1][0]), 1, yface_t, ... );
/* Send upper yface */
MPI_Send(&(arr[0][PY][0]), 1, yface_t, ... );
/* Receive lower face */
MPI_Recv(&(arr[0][0][0]), 1, yface_t, ... );
/* Receive upper face */
MPI_Recv(&(arr[0][PY+1][0]), 1, yface_t, ... );

The above is exactly the way you'd use the corresponding vector type - by pointing it at the first item to send/receive.

If you choose to use a subarray type, either way of using it is perfectly fine, and you'll see both choices made in various pieces of software. It's just a matter of which you fine clearer - 4 types per pattern (depending on the offset), or using the offset explicitly in the send/receive. I personally find the 1-type approach much clearer, but there's no unambiguous right answer to be found for that question.

As to whether to use MPI_Subarray or Vector (say), it's easiest to look at the other two patterns you'll need to support: With an X face (here you have a couple more options, since they're contiguous:

• (PY+2)*(PZ+2) MPI_Doubles
• 1 MPI_Type_Contiguous of (PY+2)*(PZ+2) MPI_Doubles
• MPI_Type_vector of count 1, blocklen (PY+2)*(PZ+2), and stride of anything, or of count PY+2, blocklen PZ+2, and stride of PZ+2, or any equivalent combination
• a subarray, with a slice subsize of [1,PY+2,PZ+2], starting at an appropriate location

and for z-faces:

• MPI_Type_vector of count (PX+2)*(PY+2), blocklen 1, and stride of PZ+2
• a subarray, with a slice subsize of [PX+2,PY+2,1], starting at an appropriate location.

So again, it all comes down to clarity. The subarray type looks the most similar between all directions, and the difference is fairly clear; whereas if I showed you a bunch of vector types all declared in the same piece of code, you'd have to do some sketching on a whiteboard to make sure I hadn't accidentally switched them around. The subarray also generalizes the most easily - if you move to a method that now needs 2 halo cells on each side, say, or to not send the corner cells, the modification to the subarray is trivial, whereas you have to do some work to build something with vectors.