性能比较MPI与OpenMP

Question

I have a very strange problem. I even do not know if I can provide you all the information you need to answer my question; in case something is missing, please let me know.

I run a code like this using MPI:

#include <mpi.h>
#include <cmath>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <string.h>
#include <cstdlib>
#include <fstream>
#include <sstream>
#include <iomanip>
#include <iostream>
#include <stdexcept>
#include <algorithm>
#include "mkl.h"
double *gradient_D = new double[K*M];
double *DX = new double[M*N];
double gradientD_time = MPI_Wtime();
for (int j = 0; j < K; j++){
    for (int i = 0; i < M; i++){
        gradient_D[j*M+i] = 0;
        for (int k = 0; k < n; k++)
            gradient_D[i+M*j] += DX[i+k*M];
        }   
    }  
double gradientD_total_time = (MPI_Wtime() - gradientD_time);
printf("Gradient D total = %f \n", gradientD_total_time);

It odes not really matter the meaning of the code: I am just running three for loops and evaulating the CPU time. In the cmake I wrote the following commands:

project(mpi_algo)
cmake_minimum_required(VERSION 2.8)
set(CMAKE_CXX_COMPILER "mpicxx")
set(CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS)
set(CMAKE_CXX_FLAGS "-cxx=icpc -mkl=sequential")
add_executable(mpi_algo main.cpp)

and I run the code:

mpirun -np 1 ./mpi_algo

After that, I run a similar code in which I do the same operations, but using OpenMP instead of MPI:

#include <omp.h>
#include <cmath>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <string.h>
#include <cstdlib>
#include <fstream>
#include <sstream>
#include <iomanip>
#include <iostream>
#include <stdexcept>
#include <algorithm>
#include "mkl.h"
double *gradient_D = new double[K*M];
double *DX = new double[M*N];
double gradientD_time = omp_get_wtime();
for (int j = 0; j < K; j++){
    for (int i = 0; i < M; i++){
        gradient_D[j*M+i] = 0;
        for (int k = 0; k < n; k++)
            gradient_D[i+M*j] += DX[i+k*M];
        }   
    }  
double gradientD_total_time = (omp_get_wtime() - gradientD_time);
printf("Gradient D total = %f \n", gradientD_total_time);

You can see that there are small differences in the code. This is the cmake:

project(openmp_algo)
cmake_minimum_required(VERSION 2.8)
set(CMAKE_CXX_COMPILER "icc")
set(CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS)
set(CMAKE_CXX_FLAGS "-qopenmp -mkl=sequential")
add_executable(openmp_algo main.cpp)

and I run the code:

./openmp_algo

Now, what I can not explain is that the code with MPI takes about 1 second to run. The other one, that should be the same, takes about 20 seconds.

Could you please someone explain me why?

EDIT: the constants M, N, n, k do not matter for understanding the issue. They just define the dimension of the arrays.

Answer 1

Since you don't give much details on the environment, I will make a wild guess to try to give an explanation. First, let's make a few remarks:

• Your seemingly identical just do nothing, so a clever compiler is fully entitled to optimize away your compute loops and just do the printing;
• Your OpenMP code is compiled with a vanilla icc (odd choice for a C++ code BTW) which optimization level will therefore be the default -O2 (minus the extra optimization seen as not thread-safe by default that using -qopenmp will disable;
• Your MPI code is compiled with a plain mpicxx which will call internally icpc as compiler.

This is the mpicxx that I suspect is the key here: indeed, mpicxx is just a wrapper to the actual compiler, which will also set some include path, some library path and list, but also might set some extra optimization options. In some cases for example, the optimization options used while installing the MPI library will be kept inside the mpicxx wrapper and silently used by default when compiling your codes...

So here is my guess, your mpicxx set among other the -O3 optimization option and therefore, the compiler will optimize away the loop for MPI, while the default -O2 that you get for your OpenMP code doesn't do it. Therefore, you're timing pretty-much nothing in the case of your MPI code, while you're timing the actual loop execution with your OpenMP one.

Just a guess, but that seems fair enough. A good test would be to check what a mpicxx -cxx=icpc -show would give you...