qemu-kvm如何在内部创建VM线程？

Question

Now I am doing a project on qemu-kvm and linux task scheduler.I know each VCPU is a normal task created by qemu to the linux OS. Then I try to execute the qemu command to see how the task is created. I use strace to track all the system calls. There are no things like "fork" or "pthreadcreate".But I have seen this:

open("/dev/kvm", O_RDWR|O_LARGEFILE)    = 3
ioctl(3, KVM_GET_API_VERSION, 0)        = 12
ioctl(3, KVM_CHECK_EXTENSION, 0x19)     = 0
ioctl(3, KVM_CREATE_VM, 0)              = 4
ioctl(3, KVM_CHECK_EXTENSION, 0x4)      = 1
ioctl(3, KVM_CHECK_EXTENSION, 0x4)      = 1
ioctl(4, KVM_SET_TSS_ADDR, 0xfffbd000)  = 0
ioctl(3, KVM_CHECK_EXTENSION, 0x25)     = 0
ioctl(3, KVM_CHECK_EXTENSION, 0xb)      = 1
ioctl(4, KVM_CREATE_PIT, 0xb)           = 0
ioctl(3, KVM_CHECK_EXTENSION, 0xf)      = 2
ioctl(3, KVM_CHECK_EXTENSION, 0x3)      = 1
ioctl(3, KVM_CHECK_EXTENSION, 0)        = 1
ioctl(4, KVM_CREATE_IRQCHIP, 0)         = 0
ioctl(3, KVM_CHECK_EXTENSION, 0x1a)     = 0

So it looks that it opens the devices /dev/kvm and did some ioctl syscalls. I believe this is the place where the VM thread is actually created. Right? I am new to the OS stuff and I will appreciate if anyone can give me some clue:> Thanks

Answer 1

VCPU is neither a OS thread nor a process. To understand how VCPU works, first we should figure out how guest OS is running on Intel VT-x architecture.

Intel VT-x proposed a new mode methodology with two modes: VMX root mode and VMX non-root mode , for running host VMM and guest respectively. Intel VT-x also contains a new structure: VMCS , which saves all information both host and guest need. VMCS is one per guest.

KVM is a hardware-assisted hypervisor and leverages Intel VT-x . The host Linux KVM is running in VMX root mode . When KVM decides to switch CPU mode to run a guest, KVM dumps all current contexts to VMCS and executes a "VMLAUNCH" instruction. "VMLAUNCH" will transfer CPU from VMX root mode to VMX non-root mode , and load guest context from VMCS, then start or continue to execute guest code.

In summary, the guest code is running directly on CPU in VMX non-root mode . no software emulation layer for VCPU is needed. That's why KVM has better performance, and there is no specific thread for guest.

/dev/kvm is created by kvm.ko , which is only a KVM interface for QEMU. Your strace output showed how QEMU was interacting with KVM and controlling the underlying guests. You can never find a fork or clone system call in KVM.

For more KVM detail especially VCPU, you can read KVM code in arch/x86/kvm/vmx.c for more VCPU implementation detail based on Intel VT-x .

Answer 2

Even though a VCPU is an OS object different from a thread or a process, and VCPU objects are created with the KVM_CREATE_VCPU ioctl, QEMU is indeed creating a thread per VCPU. The guest runs (the physical CPU enters VMX non-root mode) when QEMU does KVM_RUN from that thread. KVM_CREATE_VCPU returns a new file descriptor, and that's the fd you'll see in the KVM_RUN ioctl.

VCPU threads might be missing from your strace because you did not use the -ff option. -ff asks strace to also trace other threads than the initial one.