CPU stuck Centos 7.2 on Azure

Question

i'm running a CentOS 7.2 VM on Azure and get a CPU stuck kernel-bug warning. top shows that CPU#0 is 100% in use.

[admin@bench2 ~]$
Message from syslogd@bench2 at Feb  9 10:06:43 ...
kernel:BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u128:1:13777]

This is the top output:

Tasks: 258 total,   7 running, 251 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,100.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0  st
%Cpu1  :  0.3 us,  0.3 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 28813448 total, 26938144 free,   653860 used,  1221444 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 27557900 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
73 root      20   0       0      0      0 S   0.7  0.0   1:03.03 rcu_sched
1 root      20   0   43668   6204   3796 S   0.0  0.0   0:04.70 systemd
2 root      20   0       0      0      0 S   0.0  0.0   0:00.03 kthreadd
3 root      20   0       0      0      0 R   0.0  0.0   0:00.10 ksoftirqd/0
5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H

Centos + Kernel Version:

CentOS Linux release 7.1.1503 (Core)
Linux bench2 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

If noticed that this error also appears on CentOS 7.2 versions.

[84176.089080] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u128:1:13777]
[84176.089080] Modules linked in: vfat fat isofs xfs libcrc32c iptable_filter ip_tables udf crc_itu_t hyperv_fb hyperv_keyboard hv_utils i2c_piix4 i2c_core serio_raw pcspkr crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_common hv_netvsc hv_storvsc hid_hyperv sr_mod cdrom ata_generic pata_acpi ata_piix libata floppy hv_vmbus
[84176.089080] CPU: 0 PID: 13777 Comm: kworker/u128:1 Tainted: G        W   --------------   3.10.0-229.7.2.el7.x86_64 #1
[84176.089080] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006  05/23/2012

If this version does problems on Azure it is no problem to switch it. If this is the case, I want to know which CentOS version would be the best to run on an Azure environment.

Answer 1

I solved the problem by setting the Host caching on VHD to None . Odd behaviour but it works.

see screen here

Answer 2

I had the same issue, it's a disk performance issue (high IOPS/Latency etc.), not related to CPU or RAM (at least in my case). The storage (NetApp) was very loaded, I solve it by moving to SSD, even using a large raid group with HDD (without a special load) didn't help.

We used a K8S setup, but I saw it on lot of CentOS with simple applications as well.

Regards,