16-bit float MPI_Reduce?

Question

I have a distributed application that uses MPI_Reduce() for some of the communication. In terms of precision, we get fully accurate results with 16-bit float numbers (half-precision).

To accelerate the communication (reducing the amount of data movement), is there a way to call MPI_Reduce() on 16-bit float numbers?

---

(I looked at the MPI documentation and didn't see any info about 16-bit floats.)

Answer 1

The MPI standard defines only 32-bit ( MPI_FLOAT ) or 64-bit ( MPI_DOUBLE ) floats in its internal datatypes.

However, you can always create your own MPI_Datatype and your own custom reduce operation. The code below gives some rough idea of how you can do this. Since it is unclear which 16 bit float implementation you are using, I'm going to refer to the type simply as float16_t and the addition operation as fp16_add() .

// define custom reduce operation
void my_fp16_sum(void* invec, void* inoutvec, int *len,
              MPI_Datatype *datatype) {
    // cast invec and inoutvec to your float16 type
    float16_t* in = (float16_t)invec;
    float16_t* inout = (float16_t)inoutvec;
    for (int i = 0; i < *len; ++i) {
        // sum your 16 bit floats
        *inout = fp16_add(*in, *inout);
    }
}

// ...

//  in your code:

// create 2-byte datatype (send raw, un-interpreted bytes)
MPI_Datatype mpi_type_float16;
MPI_Type_contiguous(2, MPI_BYTE, &mpi_type_float16);
MPI_Type_commit(&mpi_type_float16);

// create user op (pass function pointer to your user function)
MPI_Op mpi_fp16sum;
MPI_Op_create(&my_fp16_sum, 1, &mpi_fp16sum);

// call MPI_Reduce using your custom reduction operation
MPI_Reduce(&fp16_val, &fp16_result, 1, mpi_type_float16, mpi_fp16sum, 0, MPI_COMM_WORLD);

// clean up (freeing of the custom MPI_Op and MPI_Datatype)
MPI_Type_free(&mpi_type_float16);
MPI_Op_free(&mpi_fp16sum);