無法在分叉進程中設置OpenMP線程關聯

Question

我試圖使用openMP在不同的CPU上運行兩個進程。 在這種情況下，每個CPU都有6個帶有超線程的內核（因此有12個硬件線程）。 他們需要做一些同步，如果他們知道彼此的PID就會更容易。 所以我使用fork()和execve()從sigS啟動進程sigC ，並為GOMP_CPU_AFFINITY環境變量調用不同的值。 在fork()/execve()調用之后， sigS仍具有正確的親和力，但sigC打印

libgomp: no cpus left for affinity setting

並且所有線程都在同一個核心上。

sigS的代碼：

#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <omp.h>
#include <sched.h>

int main( void )
{
   omp_set_num_threads(12); //12 hardware threads per CPU
   //this loop runs as expected
   #pragma omp parallel for
   for( int i = 0; i<12; i++ ) {
      #pragma omp critical 
      {
         printf("TEST PRE-FORK: I am thread %2d running on core %d\n",
                omp_get_thread_num(), sched_getcpu());
      }
   }

   pid_t childpid = fork();

   if( childpid < 0 ) {
      perror("Fork failed");
   } else {
      if( childpid == 0 ) { //<------ attempt to set affinity for child
         //change the affinity for the other process so it runs
         //on the other cpu
         char ompEnv[] = "GOMP_CPU_AFFINITY=6-11 18-23"; 
         char * const args[]    = { "./sigC", (char*)0 };
         char * const envArgs[] = { ompEnv,   (char*)0 };
         execve(args[0], args, envArgs);
         perror("Returned from execve");
         exit(1);
      } else {
         omp_set_num_threads(12);
         printf("PARENT: my pid     = %d\n", getpid());
         printf("PARENT: child pid  = %d\n", childpid);
         sleep(5); //sleep for a bit so child process prints first

         //This loop gives the same thread core/pairings as above
         //this is expected
         #pragma omp parallel for
         for( int i = 0; i < 12; i++ ) {
            #pragma omp critical
            {
               printf("PARENT: I'm thread %2d, on core %d.\n",
                      omp_get_thread_num(), sched_getcpu());
            }
         }
      }
   }
   return 0;
}

sigC的代碼只有一個omp並行for循環，但為了完整性：

#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <omp.h>
#include <sched.h>

int main( void )
{
   omp_set_num_threads(12);
   printf("CHILD: my pid     = %d\n", getpid());
   printf("CHILD: parent pid = %d\n", getppid());
   //I expect this loop to have the core pairings as I specified in execve
   //i.e thread 0 -> core 6, 1 -> 7, ... 6 -> 18, 7 -> 19 ... 11 -> 23
   #pragma omp parallel for
   for( int i = 0; i < 12; i++ ) {
      #pragma omp critical
      {
         printf("CHILD: I'm thread %2d, on core %d.\n",
                omp_get_thread_num(), sched_getcpu());
      }
   }
   return 0;
}

輸出：

$ env GOMP_CPU_AFFINITY="0-5 12-17" ./sigS

這部分是預期的

TEST PRE-FORK: I'm thread  0, on core 0.
TEST PRE-FORK: I'm thread 11, on core 17.
TEST PRE-FORK: I'm thread  5, on core 5.
TEST PRE-FORK: I'm thread  6, on core 12.
TEST PRE-FORK: I'm thread  3, on core 3.
TEST PRE-FORK: I'm thread  1, on core 1.
TEST PRE-FORK: I'm thread  8, on core 14.
TEST PRE-FORK: I'm thread 10, on core 16.
TEST PRE-FORK: I'm thread  7, on core 13.
TEST PRE-FORK: I'm thread  2, on core 2.
TEST PRE-FORK: I'm thread  4, on core 4.
TEST PRE-FORK: I'm thread  9, on core 15.
PARENT: my pid     = 11009
PARENT: child pid  = 11021

這就是問題 - 孩子中的所有線程都在核心0上運行

libgomp: no CPUs left for affinity setting
CHILD: my pid     = 11021
CHILD: parent pid = 11009
CHILD: I'm thread  1, on core 0.
CHILD: I'm thread  0, on core 0.
CHILD: I'm thread  4, on core 0.
CHILD: I'm thread  5, on core 0.
CHILD: I'm thread  6, on core 0.
CHILD: I'm thread  7, on core 0.
CHILD: I'm thread  8, on core 0.
CHILD: I'm thread  9, on core 0.
CHILD: I'm thread 10, on core 0.
CHILD: I'm thread 11, on core 0.
CHILD: I'm thread  3, on core 0.

（我省略了父線程打印，因為它與pre-fork相同）

關於我如何解決這個問題或是否是正確方法的任何想法？

Answer 1

fork() ed子進程繼承其父關聯掩碼。 libgomp將此親和力掩碼與GOMP_CPU_AFFINITY的集合GOMP_CPU_AFFINITY ，最后得到一個空集，因為兩個集都是互補的。 這種行為沒有記錄，但是看一下libgomp的源代碼libgomp證實確實如此。

解決方案是在進行execve()調用之前重置子進程的affinity mask：

if (childpid == 0) { //<------ attempt to set affinity for child
   cpu_set_t *mask;
   size_t size;
   int nrcpus = 256; // 256 CPUs should be more than enough

   // Reset the CPU affinity mask
   mask = CPU_ALLOC(nrcpus);
   size = CPU_ALLOC_SIZE(nrcpus);
   for (int i = 0; i < nrcpus; i++)
      CPU_SET_S(i, size, mask);
   if (sched_setaffinity(0, size, mask) == -1) { handle error }
   CPU_FREE(mask);

   //change the affinity for the other process so it runs
   //on the other cpu
   char ompEnv[] ="GOMP_CPU_AFFINITY=6-11 18-23"; 
   char * const args[]    = {"./sigC", (char*)0};
   char * const envArgs[] = {ompEnv,   (char*)0};
   execve(args[0], args, envArgs);
   perror("Returned from execve");
   exit(1);
} else {