CPU热插拔和严格的1：1线程

Question

I wish to add support for CPU hot plugging in my application that has strict affinity rules. Each physical core has exactly one thread pinned to it, but the logic I use for pinning a thread to a CPU is fairly naive and fails if a CPU between 0..N is offline.

I have chosen an approach where I have an array of size MAX_CPU , where each CPU on the system maps to a slot by its identifier. For example, CPU0 -> threads[0] , and CPU1 -> threads[1] , and so on. The idea is to mirror the system's setup.

for (i = 0; i < N; ++i)
    set_affinity(threads[i], i);

However, if an offline CPU is encountered anywhere but the end, it will fail.

To make things worse, when a CPU is taken offline during runtime, the pinned thread's affinity mask is reset without notice.

Ultimately, I hope to support complex setups like:

CPU0     CPU1      CPU2     CPU3
ONLINE   OFFLINE   ONLINE   OFFLINE

How can I incorporate awareness of online and offline CPU:s into my application?

I'm avoiding /proc and /sys as I'm interested in porting to other platforms, specifically various BSD:s. I use x86_64 for now, so the cpuid instruction may be useful.

Answer 1

Turns out sched_getaffinity(2) is awesome and better suited for this case. The cpu_set_t it fills in is not a generalized mask like 0xffffffff... that would imply schedule anywhere, but actually a detailed and up-to-date mask of each online and permitted CPU. Using the CPU_* macros, it's possible to extract how many CPU:s are online and which.

cpu_set_t set;
int i, n;

if (sched_getaffinity(0, sizeof(cpu_set_t), &set))
    return;

n = CPU_COUNT(&set);

/* pin a thread to each online and permitted cpu */
for (i = 0; n; ++i)
    if (CPU_ISSET(i, &set)) {
        /* spawn a thread and pin it to cpu identified by 'i' ... */
        --n;
    }

When notified of a hot plug event, a call to sched_getaffinity(2) from a non-pinned thread will give us an updated mask.

An added benefit is awareness of utilities like taskset(1) .

FreeBSD has a cpuset_getaffinity(2) system call that probably operates in a similar fashion. I don't have access to try it out now.