在被调用的core_pattern过程中调用sched_setaffinity失败

Question

I have /proc/sys/kernel/core_pattern set to pipe core dumps to my program. This program calls a shell script (using "system") which in turn calls my test program, attached. This test program attempts to call sched_setaffinity which fails, EINVAL, for any CPU. The same test program called directly from a shell script (running as root) works OK.

#include <stdio.h>      
#include <stdlib.h>     
#include <unistd.h>
#include <signal.h>
#include <errno.h>
#include <string.h>
#include <string>
#include <sys/time.h>
#include <sys/resource.h>
#include <syslog.h>


typedef unsigned long long int UINT64;

// Functions to convert Linux CPU list to/from UINT64
static UINT64 GetCpuMask(cpu_set_t* cpuSet)
{
    UINT64 mask = 0;
    const int maxCpus = CPU_SETSIZE < 64?CPU_SETSIZE:64;
    for (int cpu = 0; cpu < maxCpus; cpu++){
        mask |= CPU_ISSET(cpu, cpuSet)? 1ll<<cpu : 0;
    }
    return mask;
}
static void GetCpuSet(UINT64 mask, cpu_set_t* cpuSet)
{
    const int maxCpus = CPU_SETSIZE < 64?CPU_SETSIZE:64;
    CPU_ZERO(cpuSet);
    for (int cpu = 0; cpu < maxCpus; cpu++){
        if (1ll<<cpu & mask){
            CPU_SET(cpu, cpuSet);
        }
    }
}
UINT64 GetThreadAffinityMask()
{
    cpu_set_t cpuSet;

    if (!sched_getaffinity(0, sizeof(cpu_set_t), &cpuSet)){
        return GetCpuMask(&cpuSet);
    }
    syslog (LOG_ERR, "sched_getaffinity fails, errno %d:%s", errno, strerror(errno));
    return 0;
}

int main( int argc, char *argv[] )
{
// Temp test...
    setlogmask (LOG_UPTO (LOG_INFO)); // unless overridden in config file
    openlog ("proc3", LOG_CONS | LOG_PID | LOG_NDELAY, LOG_USER );

    UINT64 old_mask=GetThreadAffinityMask();
    syslog (LOG_INFO, "old_mask 0x%x", old_mask);

    cpu_set_t testCpuSet;
    for (int cpu = 0; cpu < 16; cpu++){
        CPU_ZERO(&testCpuSet);
        CPU_SET(cpu, &testCpuSet);
        UINT64 try_mask = GetCpuMask(&testCpuSet);
        if (!sched_setaffinity(0, sizeof(cpu_set_t), &testCpuSet)){
            syslog (LOG_INFO, "sched_setaffinity OK, cpu %d (mask 0x%x)",cpu, try_mask);
            UINT64 new_mask = GetThreadAffinityMask();
            syslog (LOG_INFO, "new_mask 0x%x", new_mask);
        } else {
            syslog (LOG_INFO, "sched_setaffinity, cpu %d errno %d:%s", cpu, errno, strerror(errno));
        }
    }
}

The logged output when called directly from a root shell:

Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: old_mask 0xf
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity OK, cpu 0 (mask 0x1)
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: new_mask 0x1
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity OK, cpu 1 (mask 0x2)
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: new_mask 0x2
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity OK, cpu 2 (mask 0x4)
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: new_mask 0x4
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity OK, cpu 3 (mask 0x8)
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: new_mask 0x8
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 4 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 5 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 6 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 7 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 8 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 9 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 10 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 11 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 12 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 13 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 14 errno 22:Invalid argument
Sep 19 11:42:43 apxdev-cjn-1 proc3[9122]: sched_setaffinity, cpu 15 errno 22:Invalid argument

The logged output when called from the dump hook:

Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: old_mask 0xf
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 0 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 1 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 2 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 3 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 4 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 5 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 6 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 7 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 8 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 9 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 10 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 11 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 12 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 13 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 14 errno 22:Invalid argument
Sep 19 11:50:25 apxdev-cjn-1 proc3[12372]: sched_setaffinity, cpu 15 errno 22:Invalid argument

Answer 1

With the addition of the kernel version, an answer:

It appears that EINVAL in the sched_setaffinity call is due to the PF_NO_SETAFFINITY flag being set in the task flags. This flag is enabled in most kernel threads to ensure that user-level programs don't manipulate the affinity of the kernel thread in inappropriate ways.

In later versions of the kernel, code was added to the flush_old_exec function in fs/exec.c causing the flag to be cleared when a user-level helper program is exec'd (such as that invoked by the piped core_pattern function here).

The source for the kernel version specified by OP is available at https://buildlogs.centos.org/c7.1511.u/kernel/20160217024115/3.10.0-327.10.1.el7.x86_64/ . The code that clears PF_SET_NOAFFINITY when exec'ing a user-mode program is not present in this version.

This flag will be inherited by child processes (AFAICT: the entire task structure is copied and I cannot see anything that would clear it).

According to https://www.kernel.org/doc/Documentation/filesystems/proc.txt , the flags value should be visible in /proc/<pid>/stat ( PF_SET_NOAFFINITY in this version is #defined to 0x04000000). You should be able to decode the values there and determine for sure whether the flag is set.