Google Compute Engine CPU usage discrepancy in Dashboard and VM

Question

In Google Cloud, I have an autscaling setup which scales depending on CPU usage. Even if no processes are consuming CPU on the VM, it is maxed out in the dashboard. Basically, the dashboard shows a much larger CPU usage than what the VM actually is using. This is triggering a lot of instance start-ups. Unnecessarily.

This is the CPU usage graph from Google Compute Engine dashboard. This is the usage for a single instance and not for the overall group.

Google Compute Engine CPU usage from dashboard

And, this is the CPU usage stats for all the CPUs from the VM. This has hardly any consumption, and most of the time is spent idle.

CPU usage from the VM

Answer 1

Is it possible that these VMs perform some heavy I/O operations? (using persistent disk, network communication). If so, this could be virtualization overhead of VMs, which is not visible to the guest operating system.

Answer 2

Look at your "CPU Usage from the VM" screenshot, the very top line, right side:

load average: 0.61, 0.94, 0.86

This is your CPU usage average over 1, 5, 15 minutes. See here for more details .

So you can see that there is something going on, and the Google Compute Engine CPU graph isn't wrong. It looks like it's graphing your 1 minute average.