What does taskset in linux exactly do?

Question

I have a program running on a 32 core system using Intel TBB. The problem I have is when I set the program to use 32 threads, the performance doesn't gain enough compared to 16 threads (only 50% boost). However, when I use:

taskset 0xFFFFFFFF ./foo

which would lock the process to 32 cores, the performance is much better.

I have the two following questions:

1. Why? By Default, the OS would use all 32 cores for the 32 thread program anyway.
2. I'm assuming that even with taskset , the OS is allowed (would) to exchange the virtual threads and the physical threads, ie threads are not pinned. am I right?

Thanks.

Answer 1

The operating system may choose to use less cores for cache purposes. Imagine if the application uses the same set of memory then each write causes a cache invalidate. Forcing the lock is essentially you telling the OS the cache overhead for concurrency is not worth it, go ahead and use all the cores.

You must also remember there are other processes to run (like kthreads from the kernel, and background processes.) and migrating threads between cores is costly and may cause imbalances if your threads are not doing an even amount of work.

Also remember that the OS tries to evenly distribute work on the cores across ALL processes not just yours. This means that the load balancer may choose to not place your process on all 32 cores as there are other processes currently running and migration costs could be high or spreading your process evenly could cause load imbalance among the cpu cores. The OS strives for best system performance not necessarily best per application performance.