在 OpenMP 并行区域内设置关联

Question

I'm trying to see if it's possible to set affinity within an OpenMP region using pthread_setaffinity_np() call, assuming the underlying implementation uses pthreads for OpenMP workers. In the example code below, the call to set affinity does not return an error and the sched_getcpu() call also confirms that core affinity has been set correctly. However, this way of setting affinity causes considerable performance degradation compared to using GOMP_CPU_AFFINITY environment variable to set affinity, which indicates some underlying issue with the use of pthread_setaffinity_np(). Are there any known issues with using pthread_setaffinity_np() inside OpenMP regions? For my use-case, I need to use pthreads that are 'masters' and each pthread will call its own OpenMP regions and will need to set affinity explicitly for the respective OpenMP regions.

#pragma omp parallel for reduction(+:sum) num_threads(num_drones)
  for (int i=start_N;i<end_N;i++){
    if(set[omp_get_thread_num()] == 0) {
      set[omp_get_thread_num()] = 1;
      cpu_set_t cpuset;
      CPU_ZERO(&cpuset);
      CPU_SET(rank*num_drones+omp_get_thread_num(), &cpuset);

      int error = pthread_setaffinity_np(pthread_self(), sizeof(cpuset), &cpuset);
      if (error != 0) {
        cout<< "\nError setting affinity";
        abort();
      }
    } else if (set[omp_get_thread_num()] == 1){
      set[omp_get_thread_num()] = 2;
      assert(rank*num_drones+omp_get_thread_num() == sched_getcpu());
    }
    sum += v1[i];
  }

Answer 1

This is a bad idea. The OpenMP runtime will almost certainly be optimising its internal algorithms and data structures based on the affinity of the threads which it has established. (Eg using a hierarchical barrier to minimise cross-cache and cross-socket communication). You are stamping all over that.

You say

I need to use pthreads that are 'masters' and each pthread will call its own OpenMP regions and will need to set affinity explicitly for the respective OpenMP regions.

yet you haven't said anything about why you believe you need to do this.

This feels very like a classic "I have a problem I'm not going to explain to you, but here's my solution that doesn't work, so please fix that solution for me" question.

If you were to explain your real problem we might be able to provide more help...

(In particular, choosing thread affinity sensibly using OpenMP's mechanisms may be all that you need. See Controlling OpenMP Thread Affinity ).