std::promise::set_value on FIFO thread pinned to a core doesn't wake std::future

Question

I am trying to create a system which has a deterministic realtime response.

I create a number of cpusets , move all non-critical tasks and unpinned kernel threads to one set, and then pin each of my realtime threads to its own cpuset, each of which consists of a single cpu.

$ non-critical tasks and unpinned kernel threads
cset proc --move --fromset=root --toset=system
cset proc --kthread --fromset=root --toset=system

$ realtime threads
cset proc --move --toset=shield/RealtimeTest1/thread1 --pid=17651
cset proc --move --toset=shield/RealtimeTest1/thread2 --pid=17654

My scenario is this:

• Thread 1: SCHED_OTHER , pinned to set1 , waiting on std::future<void>
• Thread 2: SCHED_FIFO , pinned to set2 , calls std::promise<void>::set_value()

Thread 1 blocks forever. However, if I change Thread 2 so be SCHED_OTHER , Thread 1 is able to continue.

I have run an strace -f to get more insight; it seems Thread 1 is waiting on a futex (I assume the internals of std::future ) but is never woken up.

I'm absolutely stymied - is there any way to have a thread pin itself to a core and set its scheduler to FIFO , and then use a std::promise to wake up another thread which is waiting for it to complete this so-called realtime setup?

The code for thread1 creating thread2 is as follows:

// Thread1:
std::promise<void> p;
std::future <void> f = p.get_future();

_thread = std::move(std::thread(std::bind(&Dispatcher::Run, this, std::ref(p))));

LOG_INFO << "waiting for thread2 to start" << std::endl;

if (f.valid())
    f.wait();

and the Run function for thread2 is as follows:

// Thread2:
LOG_INFO << "started: threadId=" << Thread::GetId() << std::endl;

Realtime::Service* rs = Service::Registry::Lookup<Realtime::Service>();
if (rs)
    rs->ConfigureThread(this->Name()); // this does the pinning and FIFO etc

LOG_INFO << "thread2 has started" << std::endl;
p.set_value(); // indicate fact that the thread has started

The strace output follows:

• Thread 1 is [pid 17651]
• Thread 2 is [pid 17654]

In the interests of brevity I have removed some of the output.

//////// Thread 1 creates thread 2 and waits on a future ////////

[pid 17654] gettid()                    = 17654
[pid 17651] write(2, "09:29:52 INFO waiting for thread"..., 4309:29:52 INFO waiting for thread2 to start
 <unfinished ...>
[pid 17654] gettid( <unfinished ...>
[pid 17651] <... write resumed> )       = 43
[pid 17654] <... gettid resumed> )      = 17654
[pid 17651] futex(0xd52294, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 17654] gettid()                    = 17654
[pid 17654] write(2, "09:29:52 INFO thread2 started: t"..., 6109:29:52 INFO thread2  started: threadId=17654
) = 61

//////// <snip> thread2 performs pinning, FIFO, etc </snip> ////////

[pid 17654] write(2, "09:29:52 INFO thread2 has starte"..., 3409:29:52 INFO thread2 has started
) = 34
[pid 17654] futex(0xd52294, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0xd52268, 2) = 1
[pid 17651] <... futex resumed> )       = 0
[pid 17654] futex(0xd522c4, FUTEX_WAKE_PRIVATE, 2147483647 <unfinished ...>
[pid 17651] futex(0xd52268, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 17654] <... futex resumed> )       = 0
[pid 17651] <... futex resumed> )       = 0

//////// blocks here forever ////////

You can see that pid 17651 (thread1) reports futex resumed , but is it maybe running on the wrong cpu and getting blocked behind thread2 which is running as FIFO ?

Update: It seems this is an issue with threads not running on the cpus they have been pinned to .

top -p 17649 -H with f,j to bring up the last used cpu shows that thread 1 is indeed running on thread 2's cpu .

top - 10:00:59 up 18:17,  3 users,  load average: 7.16, 7.61, 4.18
Tasks:   3 total,   2 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  7.1%us,  0.1%sy,  0.0%ni, 89.5%id,  0.0%wa,  0.0%hi,  3.3%si,  0.0%st
Mem:   8180892k total,   722800k used,  7458092k free,    43364k buffers
Swap:  8393952k total,        0k used,  8393952k free,   193324k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND                                                                                                                                                                                             
17654 root      -2   0 54080  35m 7064 R  100  0.4   5:00.77 3 RealtimeTest                                                                                                                                                                                         
17649 root      20   0 54080  35m 7064 S    0  0.4   0:00.05 2 RealtimeTest                                                                                                                                                                                         
17651 root      20   0 54080  35m 7064 R    0  0.4   0:00.00 3 RealtimeTest    

However, if I look at the cpuset filesystem, I can see that my tasks are supposedly pinned to the cpus I requested:

/cpusets/shield/RealtimeTest1 $ for i in `find -name tasks`; do echo $i; cat $i; echo "------------"; done

./thread1/tasks
17651 
------------
./main/tasks
17649 
------------
./thread2/tasks
17654
------------

Displaying the cpuset config:

$ cset set --list -r
cset: 
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root       0-23 y     0-1 y   279    2 /
       system 0,2,4,6,8,10 n       0 n   202    0 /system
       shield 1,3,5,7,9,11 n       1 n     0    2 /shield
RealtimeTest1    1,3,5,7 n       1 n     0    4 /shield/RealtimeTest1
      thread1          3 n       1 n     1    0 /shield/RealtimeTest1/thread1
      thread2          5 n       1 n     1    0 /shield/RealtimeTest1/thread2
         main          1 n       1 n     1    0 /shield/RealtimeTest1/main

From this I would say that thread2 is supposed to be on cpu 5, but top says it's running on cpu 3.

Interestingly, sched_getaffinity reports what cpuset does - that thread1 is on cpu 3 and thread2 is on cpu 5.

However, looking at /proc/17649/task to find the last_cpu each of its tasks ran on:

/proc/17649/task $  for i in `ls -1`; do cat $i/stat | awk '{print $1 " is on " $(NF - 5)}'; done
17649 is on 2
17651 is on 3
17654 is on 3

sched_getaffinity reports one thing, but reality is another

Interestingly, main thread [ pid 17649 ] is supposed to be on cpu 1 (according to the cset output), but in fact it is running on cpu 2 (which is on another socket)

So I would say that cpuset is not working?

My machine configuration is:

$ cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 1
$ uname -a
Linux foobar 2.6.32.12-0.7-default #1 SMP 2010-05-20 11:14:20 +0200 x86_64 x86_64 x86_64 GNU/Linux

Answer 1

I have reran the tests on a SLES 11 / SP 2 box, and the pinning works.

As such, I'm going to mark this as an answer, which is: This is an issue related to SP 1