Virtual Machines

01 Introduction

The previous units presented the operating system as the manager of the hardware and the protector of the system's resources: the process management (Unit 3), the CPU scheduling (Unit 5), the memory management (Units 8 and 9), the file systems (Unit 10), the I/O and the storage (Unit 11), and the security and the protection (Unit 12). This unit presents the operating system's virtualization: the abstraction that lets one physical machine run several operating systems, and that lets the cloud provide the on-demand's computing.

A virtual machine (VM) is the software's emulation of a physical computer: the VM presents the guest's operating system with the virtual hardware (the virtual CPUs, the virtual memory, the virtual devices), and the guest's operating system runs as if it were on the physical's hardware. The VM's design's central's observation: the guest's operating system does not know that it is the virtualized, and the guest's code (the kernel, the drivers, the applications) runs without the modification. The observation is the reason the virtualization is the transparent: the guest's operating system uses the same interfaces (the privileged's instructions, the device's registers, the interrupts) as on the physical's hardware, and the hypervisor's role is the interfaces' emulation.

The virtualization's benefits:

• The consolidation. The several VMs share the one physical's machine, and the hardware's utilization is the higher (the modern's servers' utilization without the virtualization is the 10–20%, and with the virtualization it is the 60–80%). The consolidation's savings: the hardware's cost, the power's consumption, the cooling, the space, and the management.
• The isolation. The VMs are the isolated: the VM's operating system's crash does not affect the other VMs, the VM's security's compromise does not spread to the other VMs (the hypervisor's isolation), and the VMs' resource's usage is the contained (the hypervisor's limits). The isolation's strength is the VMs' central's advantage over the containers (the containers share the host's kernel, and the VMs have the separate's kernels).
• The portability. The VM is the software's artifact: the VM's image (the disk's image, the configuration, the metadata) can be the moved between the physical's machines, the migrated between the data centers, and the deployed in the cloud. The portability's mechanisms: the VM's images (the OVF's, the OVA's, the QCOW2's, the VMDK's formats), the live's migration (the VM's state's transfer with the minimal's downtime), and the cloud's services (the AWS's AMIs, the Azure's VHDs, the GCP's images).
• The testing and the development. The VMs provide the sandboxed's environments: the developers can run the several operating systems (the Linux, the Windows, the macOS) on the one machine, the testers can test the software in the isolated's environments (the clean's snapshots, the reproducible's configurations), and the trainers can provide the students with the pre-configured's VMs.
• The legacy's support. The VMs can run the old's operating systems (the Windows XP, the old's Linux's distributions, the legacy's applications) on the modern's hardware, and the observation is the reason the VMs are the legacy's applications' preservation's mechanism.
• The high's availability and the disaster's recovery. The VMs can be the restarted on the other's physical's machines (the failover), the replicated to the other's sites (the disaster's recovery), and the snapshotted for the backup (the point-in-time's copies). The observation is the reason the VMs are the data center's high's availability's foundation.

The virtualization's history: the concept is the old (the IBM's mainframes' virtualization in the 1960s and the 1970s: the CP/CMS's and the VM/370's systems), and the modern's virtualization (the x86's) began in the late 1990s and the early 2000s (the VMware's Workstation and ESX, the Xen's paravirtualization, the KVM's kernel's integration, the Intel's VT-x and the AMD's AMD-V's hardware's assistance). The modern's virtualization's explosion was the x86's servers' consolidation's driver: the x86's servers' utilization's low, the hardware's cost's high, and the virtualization's benefits (the consolidation, the isolation, the portability) made the virtualization the data center's standard.

This lesson presents the virtual machines. It defines the VM and the virtualization's benefits. It presents the VM's architecture (the guest, the host, the hypervisor/VMM) and the types of virtualization (the full, the paravirtualization, the hardware-assisted). It presents the trap-and-emulate's technique and the x86's virtualization's challenge (the ring compression, the binary translation). It presents the hardware's assistance (the Intel's VT-x, the AMD's AMD-V), the VM's lifecycle, and the CPU, the memory, and the I/O's virtualization's mechanisms. It presents the virtualization's design's trade-offs and the lesson's synthesis.

02 Learning objectives

This lesson contributes to U13.1 and U13.2, and to course learning outcomes CLO1, CLO7, and CLO8.

03 Required reading

• Operating System Concepts — Chapter 18, Virtual Machines. The standard treatment of the virtualization's concept, the hypervisor's role, the types of virtualization, and the virtualization's benefits.
• OSTEP — the Virtualization chapters, particularly the discussion of the virtualization's mechanisms and the containers. OSTEP's treatment of the CPU's and the memory's virtualization is the clearest available account, and it is the basis for the worked example in this lesson.
• Tanenbaum & Bos — Modern Operating Systems, Chapter 7, Virtualization and the Cloud. A complementary treatment that develops the virtualization's mechanisms from first principles.
• Smith, J. E., & Nair, R. — The Architecture of Virtual Machines. IEEE Computer, 2005. The comprehensive survey of the virtualization's architectures and the hypervisor's designs.
• Goldberg, R. P. — Survey of Virtual Machine Research. IEEE Computer, 1974. The classical's paper on the virtualization's concept (the IBM's mainframes' experience), and the source of the trap-and-emulate's formalization.
• Popek, G. J., & Goldberg, R. P. — Formal Requirements for Virtualizable Third Generation Architectures. Communications of the ACM, 1974. The paper that formalized the virtualization's requirements (the instruction's classes, the trap-and-emulate's conditions).
• Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., & Warfield, A. — Xen and the Art of Virtualization. SOSP, 2003. The paper that introduced the Xen's paravirtualization.
• Kivity, A., Kamay, Y., Laor, D., Lublin, U., & Liguori, A. — KVM: the Linux Virtual Machine Monitor. Ottawa Linux Symposium, 2007. The paper that describes the KVM's design.
• Adams, K., & Agesen, O. — A Comparison of Software and Hardware Techniques for x86 Virtualization. ASPLOS, 2006. The paper that compared the binary translation and the hardware's assistance for the x86's virtualization.
• Uhlig, R., Neiger, G., Rodgers, D., Santoni, A. L., Martins, F. C. M., Anderson, A. V., Bennett, S. M., Kagi, A., Leung, F. H., & Smith, L. — Intel Virtualization Technology. IEEE Computer, 2005. The paper that describes the Intel's VT-x.
• Linux man pages: man 1 virsh, man 8 virt-install, man 1 qemu-system-x86_64 (or the QEMU's documentation), man 8 libvirtd, man 5 libvirt (or the libvirt's documentation), man 1 virt-manager, man 1 virt-clone, man 1 virt-sysprep, man 1 guestfish, man 8 kvm (or the KVM's documentation), man 1 lscpu, man 1 lspci, man 8 modprobe, man 5 qemu, man 5 qcow2 (or the QCOW2's documentation), man 8 qemu-img, man 1 vgcreate, man 1 lvcreate, man 8 brctl (or the bridge's utilities), man 8 ip, man 8 nft, man 1 dmesg, man 5 proc (for the /proc/cpuinfo's virtualization's flags), man 1 systemd-detect-virt.

04 Core content

4.1 What a virtual machine is

A virtual machine (VM) is the software's emulation of a physical computer: the VM presents the guest's operating system with the virtual hardware (the virtual CPUs, the virtual memory, the virtual devices), and the guest's operating system runs as if it were on the physical's hardware. The VM's design's central's observation: the guest's operating system does not know that it is the virtualized, and the guest's code runs without the modification.

The VM's components:

The guest

The operating system and the applications that run inside the VM. The guest's operating system is the unmodified (in the full virtualization) or the modified (in the paravirtualization), and the guest's applications run on the guest's operating system's interfaces without the knowledge of the virtualization.

The host

The physical machine and its operating system (in the hosted's virtualization) or the hypervisor (in the bare-metal's virtualization). The host provides the physical's resources (the CPUs, the memory, the storage, the network) that the VMs share, and the host's software (the hypervisor, the VMM) performs the resources' multiplexing and the virtual hardware's emulation.

The hypervisor (the VMM)

The software that creates and runs the VMs. The hypervisor's roles: the virtual hardware's emulation (the CPU's, the memory's, the devices' emulation), the resources' multiplexing (the physical's CPUs' scheduling, the memory's allocation, the devices' sharing), and the VM's lifecycle's management (the creation, the start, the stop, the migration). The hypervisor's types: the type 1 (the bare-metal's: the hypervisor runs directly on the hardware, and the guest's operating systems run on the hypervisor: the VMware's ESXi, the Xen, the Microsoft's Hyper-V's hypervisor) and the type 2 (the hosted's: the hypervisor runs as the application on the host's operating system, and the guest's operating systems run on the hypervisor: the VMware's Workstation and Fusion, the Oracle's VirtualBox, the QEMU's user's mode). The type 0 (the hybrid's: the hypervisor and the kernel are the integrated: the KVM, the Linux's kernel's virtualization's module).

The VM's benefits (the introduction's list): the consolidation, the isolation, the portability, the testing and the development, the legacy's support, and the high's availability and the disaster's recovery. The benefits' design's observation: the virtualization's value is the decoupling of the software from the hardware, and the decoupling's benefits (the flexibility, the efficiency, the resilience) are the reasons the virtualization has become the data center's and the cloud's fundamental's abstraction.

4.2 The types of virtualization

The virtualization's types differ in the technique that the hypervisor uses to virtualize the hardware and in the modifications that the guest's operating system requires:

Full virtualization

The hypervisor emulates the complete's hardware, and the guest's operating system runs unmodified. The guest's privileged's instructions are the trapped (the CPU's trap's mechanism) and the emulated (the hypervisor performs the instructions' effects on the virtual hardware), and the guest's operating system does not know that it is the virtualized. The full virtualization's advantages: the compatibility (the unmodified's guests: any x86's operating system can run), the portability (the VM's images can be the moved between the hypervisors), and the simplicity (the guest's operating system's maintenance is the standard). The full virtualization's disadvantages: the performance (the traps and the emulation's overhead: the privileged's instructions are the frequent in the guest's kernel, and the traps' cost is the significant), and the x86's virtualization's difficulty (the x86's architecture's privileged's instructions are not the all's trapped: the virtualization's hole, the section 4.4's discussion). The full virtualization's examples: the VMware's ESX's and the Workstation's (the binary translation's technique), the KVM's (the hardware's assistance), the QEMU's (the emulation's technique).

Paravirtualization

The hypervisor does not emulate the complete's hardware; instead, it presents the guest's operating system with the modified interfaces (the hypercalls: the virtualization-aware's system calls), and the guest's operating system is the modified to use the interfaces. The paravirtualization's advantages: the performance (the hypercalls are the efficient: no traps and no emulation), the simplicity (the hypervisor's design is the simpler: no binary translation, no hardware's assistance), and the scalability (the paravirtualization's overhead is the lower for the high's consolidation). The paravirtualization's disadvantages: the compatibility (the guest's operating system must be the modified: the unmodified's operating systems (the Windows) cannot run), and the maintenance (the guest's modifications must be the maintained as the operating system's versions evolve). The paravirtualization's examples: the Xen's (the classical's paravirtualization: the Linux's Xen's guest's kernel), the VMware's (the VMI: the Virtual Machine Interface), and the KVM's (the virtio: the paravirtualized's drivers for the I/O).

Hardware-assisted virtualization

The CPU's hardware provides the virtualization's support: the new's instructions and the new's modes (the Intel's VT-x's VMX's root and non-root's modes, the AMD's AMD-V's SVM's host and guest's modes) let the hypervisor run the guest's operating system in the non-root's (or the guest's) mode, and the guest's privileged's instructions are the trapped by the hardware (the VMRUN's and the VMMCALL's instructions). The hardware's assistance's advantages: the performance (the traps and the emulation are the hardware's, and the overhead is the lower), the compatibility (the unmodified's guests can run), and the simplicity (the hypervisor's design is the simpler: no binary translation, no paravirtualization's modifications). The hardware's assistance's disadvantages: the hardware's dependency (the old's CPUs do not support the virtualization's extensions), and the complexity (the hardware's virtualization's mechanisms: the EPT's and the NPT's for the memory, the VT-d's and the AMD-Vi's for the I/O). The hardware's assistance's examples: the KVM's (the default's technique on the modern's CPUs), the VMware's ESXi's (the hardware's assistance's mode), the Xen's (the HVM's mode), and the Hyper-V's.

The types' comparison:

4.3 The trap-and-emulate technique

The trap-and-emulate is the virtualization's classical's technique: the guest's operating system runs in the user's mode (the de-privileged's mode), and its privileged's instructions (the instructions that only the kernel's mode should execute) cause the CPU's trap (the exception), which the hypervisor handles and emulates (the hypervisor performs the instruction's effect on the virtual hardware, and then returns the control to the guest). The technique's design's requirements (the Popek and Goldberg's formalization):

• The instruction's classes. The instructions are classified into the privileged (the instructions that trap in the user's mode: the I/O's instructions, the page's table's manipulations, the interrupt's management), the sensitive (the instructions that read or write the privileged's state: the control's registers, the page's tables, the interrupt's flags), and the innocuous (the instructions that do neither: the arithmetic, the logic, the ordinary's memory's accesses).
• The virtualizable's condition. The architecture is virtualizable if the sensitive's instructions are the privileged's subset: the every sensitive's instruction is the trapped in the user's mode, so the hypervisor can emulate it. The condition's violation is the virtualization's hole: the sensitive's instructions that are not the privileged's (they do not trap), and the guest's operating system can execute them directly, revealing or modifying the hypervisor's state.
• The emulation's design. The hypervisor's trap's handler: it inspects the trapped's instruction, determines the instruction's type, performs the emulation (the instruction's effect on the virtual hardware), and returns the control to the guest (the iret's instruction). The emulation's cost: the trap's overhead (the mode's switch, the state's save and restore), and the emulation's logic (the instruction's decoding and the effect's computation).

The trap-and-emulate's design's observations:

• The technique's origin. The trap-and-emulate was the IBM's mainframes' technique (the 1960s and the 1970s: the IBM's System/370's and the VM/370's systems), and the technique was the foundation of the classical's virtualization.
• The technique's limits. The technique works on the architectures where the sensitive's instructions are the privileged's subset (the IBM's mainframes, the SPARC, the PowerPC), and it does not work on the architectures where the sensitive's instructions are not the privileged's (the x86, the section 4.4's discussion).
• The technique's performance. The trap-and-emulate's overhead: the every privileged's instruction's trap (the thousands of the cycles), and the emulation's logic (the instruction's decoding and the effect's computation). The observation is the reason the modern's hypervisors use the hardware's assistance (the VT-x, the AMD-V) for the CPU's virtualization, and the trap-and-emulate is the fallback for the architectures that lack the hardware's support.

4.4 The x86's virtualization's challenge

The x86's architecture's virtualization was the classical's challenge: the x86's sensitive's instructions are not the privileged's subset, and the trap-and-emulate's technique does not work. The virtualization's holes: the instructions that read or write the privileged's state but do not trap in the user's mode:

• The popf's and the pushf's instructions. The instructions that read and write the flags' register (the EFLAGS), including the interrupt's flag (the IF) and the I/O's privilege's level (the IOPL). In the user's mode, the popf's instruction silently ignores the privileged's flags (the IF, the IOPL) rather than the trap, so the guest's operating system's attempt to disable the interrupts (the cli's instruction) does not trap, and the hypervisor cannot know that the guest's interrupts' state changed.
• The segment's registers' instructions. The instructions that read and write the segment's registers (the CS, the DS, the SS) and the descriptor's tables (the GDT, the LDT, the IDT). The instructions do not trap in the user's mode, so the guest's operating system's segment's manipulations are the invisible to the hypervisor.
• The sidt, the sgdt, the sldt's instructions. The instructions that read the descriptor's tables' registers (IDTR, GDTR, LDTR). The instructions do not trap, so the guest's operating system can read the hypervisor's descriptor's tables' addresses, and the guest can determine that it is the virtualized (the sidt's instruction's result reveals the hypervisor's IDT's location).
• The iret's instruction. The instruction that returns from the interrupt's handler and can change the privilege's level. The instruction's behavior in the user's mode is the complex: the guest's operating system's use of the iret can cause the unexpected's effects.

The x86's virtualization's solutions:

The binary translation (the VMware's approach)

The hypervisor scans the guest's kernel's code (the binary's translation) and replaces the problematic's instructions with the equivalent's sequences (the traps or the calls to the hypervisor's emulation's routines). The binary translation's advantages: the unmodified's guest (the guest's operating system does not know that it is the virtualized), and the correctness (the translation's logic handles the instructions' semantics). The binary translation's disadvantages: the complexity (the x86's instruction's set is the complex, and the translation's logic is the hard), and the performance (the translation's overhead: the code's scanning and the translation's cache).

The paravirtualization (the Xen's approach)

The guest's operating system is the modified to use the hypercalls instead of the problematic's instructions: the guest's kernel's cli's and sti's instructions are the replaced with the hypercalls that the hypervisor handles. The paravirtualization's advantages: the performance (the hypercalls are the efficient: no traps and no translation), and the simplicity (the hypervisor's design is the simpler). The paravirtualization's disadvantages: the compatibility (the guest's operating system must be the modified: the unmodified's operating systems (the Windows) cannot run as the paravirtualized's guests), and the maintenance (the guest's modifications must be the maintained).

The hardware's assistance (the Intel's VT-x, the AMD's AMD-V)

The CPU's hardware provides the virtualization's support: the new's instructions and the new's modes let the hypervisor run the guest's operating system in the non-root's (or the guest's) mode, and the problematic's instructions are the trapped by the hardware. The hardware's assistance's advantages: the performance (the traps and the emulation are the hardware's), the compatibility (the unmodified's guests can run), and the simplicity (the hypervisor's design is the simpler). The hardware's assistance's mechanisms: the Intel's VT-x's VMX's root and non-root's modes, the VMCS (the Virtual Machine Control Structure: the guest's and the host's state's save and restore's area), the VMLAUNCH's and the VMRESUME's instructions (the guest's entry), the VMEXIT's (the guest's exit to the hypervisor), and the AMD's AMD-V's SVM's equivalent mechanisms (the host and the guest's modes, the VMCB, the VMRUN's instruction, the #VMEXIT's).

4.5 The CPU's virtualization

The CPU's virtualization presents the guest's operating system with the virtual CPUs (the vCPUs), and the hypervisor's scheduler multiplexes the physical's CPUs among the vCPUs:

• The vCPU's model. The vCPU is the software's abstraction of a physical's CPU: the vCPU has its own's registers (the general's registers, the control's registers, the flags), its own's execution's state (the running, the halted, the waiting), and its own's interrupts (the virtual's interrupts that the hypervisor injects).
• The vCPU's scheduling. The hypervisor's scheduler (the KVM's: the Linux's scheduler; the Xen's: the credit's scheduler and the others) allocates the physical's CPU time among the vCPUs: the vCPU is the schedulable's entity, and its scheduling's policies (the shares, the priorities, the affinities) determine the vCPU's performance. The scheduling's subtleties: the vCPU's co-scheduling (the SMP's guests' vCPUs should be the co-scheduled to avoid the spinlock's waiting), the vCPU's migration (the vCPU can be the migrated between the physical's CPUs), and the vCPU's over-commitment (the hypervisor can run more vCPUs than the physical's CPUs: the over-commitment's ratio determines the consolidation's density and the performance's variability).
• The vCPU's traps and the emulation. The guest's privileged's instructions (the I/O's instructions, the page's table's manipulations, the interrupt's management) are the trapped (the section 4.3's trap-and-emulate or the section 4.4's hardware's assistance), and the hypervisor emulates the instructions' effects on the virtual hardware. The observation: the guest's sensitive's instructions are the virtualization's overhead's sources, and the modern's hypervisors minimize the traps (the hardware's assistance, the paravirtualized's interfaces, the passthrough's devices).

4.6 The memory's virtualization

The memory's virtualization presents the guest's operating system with the virtual memory (the guest's physical's memory), and the hypervisor's memory's manager maps the guest's physical's memory to the host's physical's memory:

• The address's translation's levels. The virtualization's address's translation has the three levels: the guest's virtual's address (the GVA: the address the guest's applications use), the guest's physical's address (the GPA: the address the guest's operating system thinks is the physical's), and the host's physical's address (the HPA: the address the physical's hardware uses). The guest's page's tables translate the GVA to the GPA, and the hypervisor's mapping (the second's level of the address's translation) translates the GPA to the HPA.
• The shadow page tables (the software's approach). The hypervisor maintains the shadow page tables: the combined's page's tables that translate the GVA directly to the HPA. The hypervisor's shadow's page's tables are the synchronized with the guest's page's tables: the guest's modifications to its page's tables are the trapped (the write's protection on the guest's page's tables), and the hypervisor updates the shadow's page's tables. The shadow page tables' advantages: the translation's correctness on the CPUs without the hardware's support; the disadvantages: the complexity (the synchronization's logic) and the performance (the traps and the synchronization's overhead: the guest's page's tables' modifications are the frequent in the modern's operating systems).
• The EPT and the NPT (the hardware's approach). The Intel's EPT (the Extended Page Tables) and the AMD's NPT (the Nested Page Tables) provide the second's level's address's translation in the hardware: the guest's page's tables translate the GVA to the GPA, and the EPT's or the NPT's tables translate the GPA to the HPA. The hardware's approach's advantages: the performance (the translation is the hardware's: no traps, no shadow's page's tables' synchronization), and the simplicity (the hypervisor's memory's management is the simpler). The hardware's approach's mechanisms: the EPT's and the NPT's tables (the per-VM's tables that the hypervisor manages), the TLB's tags (the VPID's or the ASID's: the TLB's entries' VM's identification, so the VM's switch does not flush the TLB), and the hardware's page's walks (the two's levels: the guest's page's tables and the EPT's or the NPT's tables).
• The memory's ballooning. The memory's ballooning is the hypervisor's technique for the memory's over-commitment's management: the hypervisor runs the balloon's driver inside the guest's operating system, and the balloon's driver allocates the guest's memory (the balloon's inflation) and returns it to the hypervisor (the balloon's deflation). The ballooning's purpose: the hypervisor can reclaim the guest's unused's memory for the other VMs (the over-commitment's management), and the observation is the reason the ballooning is the hypervisors' memory's over-commitment's standard's mechanism.
• The page's sharing and the deduplication. The hypervisors can share the identical's memory's pages between the VMs (the KSM: the Kernel Same-page Merging: the pages' contents' comparison and the copy-on-write's sharing), and the observation: the page's sharing reduces the memory's consumption (the same's operating systems' and the applications' pages are the shared), but it introduces the security's concerns (the side channels: the shared's pages' timing's differences can leak the information, and the observation is the reason the high-security's environments disable the KSM).

4.7 The I/O's virtualization

The I/O's virtualization presents the guest's operating system with the virtual devices (the virtual's disks, the virtual's network's interfaces, the virtual's GPUs), and the hypervisor's I/O's manager routes the guest's I/O's requests to the physical's devices:

• The emulated's devices (the full's emulation). The hypervisor emulates the complete's device (the registers, the commands, the interrupts), and the guest's unmodified's driver interacts with the emulated's device as if it were the physical's. The emulation's advantages: the compatibility (the unmodified's guests: the standard's drivers work). The emulation's disadvantages: the performance (the emulation's overhead: the traps, the commands' decoding, the data's copying), and the complexity (the every's device's emulation).
• The paravirtualized's drivers (the virtio). The guest's operating system uses the paravirtualized's drivers (the virtio's drivers: the virtio-net's, the virtio-blk's, the virtio-scsi's, the virtio-gpu's), and the drivers interact with the hypervisor through the efficient's interfaces (the virtio's queues: the ring's buffers, the descriptor's tables, the notifications). The virtio's advantages: the performance (the efficient's interfaces: no traps, no emulation, the batched's requests), and the simplicity (the standard's interfaces: the guest's drivers are the same across the hypervisors). The virtio's design: the virtio's specification (the OASIS's standard), the virtio's queues (the split's virtqueue and the packed's virtqueue), and the device's types (the virtio-net, the virtio-blk, the virtio-scsi, the virtio-gpu, the virtio-fs, the virtio-vsock).
• The device's assignment (the passthrough). The hypervisor assigns the physical's device directly to the guest's VM (the VFIO's framework in Linux, the PCI's passthrough), and the guest's driver interacts with the physical's device as if it were the exclusive's owner. The passthrough's advantages: the performance (the near-native's: the guest's I/O bypasses the hypervisor), and the compatibility (the device's native's drivers work). The passthrough's disadvantages: the exclusivity (the device is the one's VM's: the other VMs cannot use it), and the migration's difficulty (the VM's live's migration is the harder when the device is the assigned).
• The SR-IOV (the Single Root I/O Virtualization). The SR-IOV's devices (the network's adapters, the storage's controllers) present the multiple's virtual functions (the VFs), and the hypervisor assigns the VFs to the different VMs. The SR-IOV's advantages: the performance (the VFs are the hardware's: the near-native's I/O), and the sharing (the multiple's VMs can use the same's physical's device through the different's VFs). The SR-IOV's disadvantages: the hardware's support (the device must support the SR-IOV), and the migration's difficulty (the VFs' assignment complicates the VM's migration).
• The vhost and the vhost-user. The KVM's vhost's framework: the hypervisor's kernel's module (the vhost-net, the vhost-scsi) handles the virtio's queues in the kernel, so the guest's I/O's requests are the processed without the user-space's (the QEMU's) involvement. The vhost-user's framework: the user-space's daemon (the virtiofsd, the OVS's vhost-user) handles the virtio's queues, so the I/O's processing is the user-space's (the flexibility's and the isolation's benefits). The observation: the vhost's and the vhost-user's frameworks are the modern's hypervisors' I/O's performance's mechanisms, and they are the virtio's design's extensions.

4.8 The VM's lifecycle

The VM's lifecycle's stages:

1. The VM's creation. The hypervisor creates the VM's configuration (the vCPUs' count, the memory's size, the devices, the disk's images, the network's interfaces), and the VM's state is the defined (the configuration exists, and the VM is not the running).
2. The VM's start. The hypervisor allocates the VM's resources (the vCPUs, the memory, the devices), the guest's operating system boots (the guest's boot's sequence: the firmware, the boot's loader, the kernel, the init), and the VM's state is the running.
3. The VM's execution. The guest's operating system and the applications run, and the hypervisor multiplexes the physical's resources among the VMs. The VM's state can be the paused (the vCPUs are the stopped, and the state is the preserved), the saved (the VM's state is the written to the disk, and the resources are the released), and the restored (the state is the read from the disk, and the VM resumes).
4. The VM's live's migration. The VM's state (the memory, the vCPUs' registers, the devices' state) is the transferred to the other's host with the minimal's downtime (the guest's operating system is the paused for the transfer's final's phase, the stop-and-copy's phase). The live's migration's mechanisms: the pre-copy's phase (the memory's pages are the copied while the VM runs), the stop-and-copy's phase (the VM is the paused, and the remaining's pages and the state are the copied), and the post-copy's phase (the VM resumes on the destination, and the remaining's pages are the fetched on the demand). The live's migration's design's observations: the migration's downtime (the stop-and-copy's phase's duration: the hundreds of the milliseconds for the typical's VMs), the migration's bandwidth (the network's bandwidth's requirement: the memory's size and the dirty's pages' rate), and the migration's compatibility (the CPUs' features, the devices' models, the hypervisors' versions).
5. The VM's stop. The hypervisor stops the VM (the graceful's shutdown: the guest's operating system's shutdown; the forced's stop: the VM's power-off), and the VM's state is the stopped. The VM's state can be the destroyed (the resources are the released, and the configuration is the removed), and the observation: the VM's disk's images and the configuration's files are the persistent's artifacts, and the VM can be the recreated from them.

The VM's lifecycle's management's tools: the virsh's and the virt-install's (the libvirt's tools: the virsh define, the virsh start, the virsh shutdown, the virsh destroy, the virsh migrate, the virsh snapshot-create, the virsh save, the virsh restore), the QEMU's qemu-system-x86_64 (the direct's invocation), the VMware's vim-cmd and the govc (the vSphere's tools), the Hyper-V's PowerShell's cmdlets, and the cloud's APIs (the AWS's EC2's API, the Azure's Resource Manager, the GCP's Compute Engine's API).

4.9 The design's trade-offs

The virtualization's design's trade-offs:

• The performance's trade-off. The virtualization's overhead's sources: the CPU's traps and the emulation (the sensitive's instructions), the memory's address's translation (the shadow's page's tables or the EPT's and the NPT's walks), the I/O's emulation (the traps, the data's copies, the interrupts' injection), and the scheduling (the vCPUs' multiplexing). The modern's hypervisors' performance's mechanisms: the hardware's assistance (the VT-x, the AMD-V, the EPT, the NPT), the paravirtualized's drivers (the virtio), the passthrough's devices (the VFIO, the SR-IOV), and the vhost's framework. The observation: the virtualization's overhead is the 1–5% for the CPU's and the memory's intensive's workloads, and it can be the higher (the 10–30%) for the I/O's intensive's workloads without the paravirtualized's drivers.
• The security's trade-off. The virtualization's security's benefits: the VMs' isolation (the separate's kernels), the snapshots (the state's rollback), and the introspection (the hypervisor's monitoring of the guest's behavior). The virtualization's security's concerns: the hypervisor's attack's surface (the hypervisor's vulnerabilities are the escapes: the VENOM's, the Xen's bugs, the KVM's vulnerabilities), the side channels (the shared's resources: the CPU's caches, the memory's deduplication, the timing's channels), and the VM's escapes (the guest's vulnerabilities that the attacker can exploit to escape the VM). The observation: the virtualization's security depends on the hypervisor's security, and the observation is the reason the high-security's environments use the minimal's hypervisors (the Xen's, the seL4's verified's microkernel) and the hardware's assistance (the IOMMU's isolation, the TPM's measured's boot).
• The compatibility's trade-off. The virtualization's compatibility's spectrum: the full virtualization (the any's guest: the unmodified's operating systems) at the performance's cost; the paravirtualization (the modified's guest: the paravirtualized's operating systems) at the compatibility's cost; the hardware's assistance (the any's guest on the modern's CPUs) at the hardware's dependency's cost. The modern's hypervisors combine the approaches (the hardware's assistance with the virtio's optional's drivers), and the observation is the reason the modern's virtualization has the both: the compatibility's and the performance's.
• The management's trade-off. The virtualization's management's concerns: the VMs' sprawl (the many VMs, the many images, the many configurations), the resources' over-commitment (the vCPUs', the memory's, the storage's), the performance's isolation (the noisy's neighbor: one VM's I/O or CPU's intensive workload affects the other VMs), and the management's complexity (the VMs' lifecycle, the images' updates, the monitoring). The management's mechanisms: the vCenter's and the oVirt's (the enterprise's management's platforms), the OpenStack's (the cloud's platform), the Kubernetes's (the container's orchestration: the Unit 13's Lesson 5), and the cloud's provider's tools (the AWS's, the Azure's, the GCP's).

4.10 Common misconceptions

05 Worked example

The following example traces a VM's execution through the hypervisor's mechanisms, computes the virtualization's overhead, and demonstrates the CPU, the memory, and the I/O's virtualization's behaviors.

# Check the virtualization's support
lscpu | grep -E 'Virtualization|Hypervisor'
grep -E 'vmx|svm' /proc/cpuinfo | head -2
lsmod | grep -E 'kvm|vfio'

# Check the libvirt's status
systemctl status libvirtd 2>/dev/null | head -5
virsh list --all 2>/dev/null

# Create a VM with the virt-install's tool
sudo virt-install \
    --name demo-vm \
    --memory 2048 \
    --vcpus 2 \
    --cpu host-passthrough \
    --disk size=10,format=qcow2,bus=virtio \
    --network network=default,model=virtio \
    --graphics none \
    --console pty,target_type=serial \
    --cdrom /path/to/installer.iso \
    --os-variant ubuntu22.04 \
    --noautoconsole

# Inspect the VM's configuration
virsh dumpxml demo-vm | head -60

# Show the VM's details
virsh dominfo demo-vm
virsh vcpuinfo demo-vm
virsh domblklist demo-vm
virsh domiflist demo-vm

# The CPU's overhead (the native versus the guest)
# Native:
time sysbench cpu --cpu-max-prime=20000 run 2>/dev/null | grep 'events per second'

# Guest (inside the VM):
virsh console demo-vm
# (inside the guest):
time sysbench cpu --cpu-max-prime=20000 run 2>/dev/null | grep 'events per second'

# The memory's overhead
# Native:
sysbench memory --memory-block-size=1M --memory-total-size=10G run 2>/dev/null | grep 'transferred'

# Guest (inside the VM):
sysbench memory --memory-block-size=1M --memory-total-size=10G run 2>/dev/null | grep 'transferred'

# The I/O's overhead (the sequential's and the random's)
# Native:
fio --name=seqread --filename=/tmp/fio-test --size=1G --rw=read --bs=1M --direct=1 --runtime=10 2>/dev/null | grep bw
fio --name=randread --filename=/tmp/fio-test --size=1G --rw=randread --bs=4k --iodepth=64 --direct=1 --runtime=10 2>/dev/null | grep bw

# Guest (inside the VM):
fio --name=seqread --filename=/tmp/fio-test --size=1G --rw=read --bs=1M --direct=1 --runtime=10 2>/dev/null | grep bw
fio --name=randread --filename=/tmp/fio-test --size=1G --rw=randread --bs=4k --iodepth=64 --direct=1 --runtime=10 2>/dev/null | grep bw

# Clean up
rm /tmp/fio-test 2>/dev/null

# The live's migration to the other's host
# (the destination's host must have the libvirt's
# daemon and the compatible's configuration)
virsh migrate --live --verbose demo-vm \
    qemu+ssh://dest-host/system \
    --unsafe --persistent

# The migration's progress is the shown in the verbose's output.

# The migration's states:
virsh list --all
virsh domjobinfo demo-vm

# The migration's parameters:
#   --live: the live's migration (the minimal's downtime)
#   --verbose: the progress's display
#   --persistent: the VM's definition's transfer
#   --unsafe: the migration without the safety's checks
#              (the incompatible's configurations)

# The post-copy's migration (the alternative's technique):
virsh migrate --live --postcopy --verbose demo-vm \
    qemu+ssh://dest-host/system

# The migration's tuning:
virsh migrate-setspeed demo-vm --bandwidth 1000  # the MB/s

# The VM's state
virsh list --all
virsh dominfo demo-vm
virsh vcpuinfo demo-vm
virsh dommemstat demo-vm

# The VM's devices
virsh domblklist demo-vm
virsh domiflist demo-vm
virsh domblkinfo demo-vm vda

# The VM's snapshots
virsh snapshot-list demo-vm
virsh snapshot-create-as demo-vm snap1 --description "The test's snapshot"
virsh snapshot-list demo-vm
virsh snapshot-info demo-vm snap1

# The VM's console
virsh console demo-vm

# The hypervisor's state
ls /sys/module/kvm/
cat /sys/module/kvm/parameters/* 2>/dev/null | head -10
cat /sys/module/kvm_intel/parameters/* 2>/dev/null | head -10

# The VM's cgroup's state
systemd-cgls /machine.slice/ 2>/dev/null
ls /sys/fs/cgroup/machine.slice/ 2>/dev/null

# The virtualization's detection
systemd-detect-virt
cat /proc/cpuinfo | grep -E 'vmx|svm' | head -2
lscpu | grep -E 'Virtualization|Hypervisor'

06 Interactive activity

07 Knowledge check

08 Unit discussion

09 Practice exercise

10 Summary

• A virtual machine (VM) is the software's emulation of a physical computer: the VM presents the guest's operating system with the virtual hardware, and the guest's operating system runs as if it were on the physical's hardware. The virtualization's benefits: the consolidation, the isolation, the portability, the testing and the development, the legacy's support, and the high's availability and the disaster's recovery.
• The VM's architecture's components: the guest (the operating system and the applications), the host (the physical machine and its software), and the hypervisor (the VMM) (the software that creates and runs the VMs). The hypervisor's types: the type 1 (the bare-metal's: the ESXi, the Xen, the Hyper-V), the type 2 (the hosted's: the Workstation, the VirtualBox, the QEMU's user's mode), and the type 0 (the hybrid's: the KVM).
• The three types of virtualization: the full virtualization (the hypervisor emulates the complete's hardware, and the guest's operating system runs unmodified: the VMware's ESX, the KVM with the hardware's assistance), the paravirtualization (the hypervisor presents the modified's interfaces (the hypercalls), and the guest's operating system is the modified: the Xen's paravirtualization, the virtio), and the hardware-assisted virtualization (the CPU's virtualization's extensions: the Intel's VT-x, the AMD's AMD-V).
• The trap-and-emulate is the virtualization's classical's technique: the guest's privileged's instructions trap, and the hypervisor emulates them. The technique's requirements (the Popek and Goldberg's formalization): the instruction's classes (the privileged, the sensitive, the innocuous) and the virtualizable's condition (the sensitive's instructions are the privileged's subset). The x86's virtualization's challenge: the virtualization's holes (the popf's, the segment's registers', the sidt's, the iret's instructions), and the solutions (the binary translation, the paravirtualization, the hardware's assistance).
• The CPU's virtualization presents the vCPUs, and the hypervisor's scheduler multiplexes the physical's CPUs. The memory's virtualization presents the guest's physical's memory, and the hypervisor's memory's manager maps the GPA to the HPA: the shadow page tables (the software's approach) or the EPT and the NPT (the hardware's approach), with the ballooning and the page's sharing for the memory's over-commitment's management. The I/O's virtualization presents the virtual devices: the emulated's devices, the paravirtualized's drivers (the virtio), the device's assignment (the passthrough), and the SR-IOV.
• The VM's lifecycle: the creation, the start, the execution, the live's migration (the pre-copy, the stop-and-copy, the post-copy), and the stop. The lifecycle's management's tools: the virsh, the virt-install, the qemu-system-x86_64, the vSphere's tools, the Hyper-V's cmdlets, and the cloud's APIs.
• The design's trade-offs: the performance's trade-off (the overhead's sources and the performance's mechanisms: the hardware's assistance, the paravirtualized's drivers, the passthrough's devices; the overhead: the 1–5% for the CPU's and the memory's, the 10–30% for the I/O's without the virtio), the security's trade-off (the VMs' isolation and the hypervisor's attack's surface, the side channels, the VM's escapes), the compatibility's trade-off (the full virtualization, the paravirtualization, the hardware's assistance), and the management's trade-off (the VMs' sprawl, the over-commitment, the noisy's neighbor, the management's complexity).

11 References

• Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 18.
• Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, the Virtualization chapters. Available online at ostep.org.
• Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 7.
• Smith, J. E., & Nair, R. The Architecture of Virtual Machines. IEEE Computer, 2005.
• Goldberg, R. P. Survey of Virtual Machine Research. IEEE Computer, 1974.
• Popek, G. J., & Goldberg, R. P. Formal Requirements for Virtualizable Third Generation Architectures. Communications of the ACM, 1974.
• Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., & Warfield, A. Xen and the Art of Virtualization. SOSP, 2003.
• Kivity, A., Kamay, Y., Laor, D., Lublin, U., & Liguori, A. KVM: the Linux Virtual Machine Monitor. Ottawa Linux Symposium, 2007.
• Adams, K., & Agesen, O. A Comparison of Software and Hardware Techniques for x86 Virtualization. ASPLOS, 2006.
• Uhlig, R., Neiger, G., Rodgers, D., Santoni, A. L., Martins, F. C. M., Anderson, A. V., Bennett, S. M., Kagi, A., Leung, F. H., & Smith, L. Intel Virtualization Technology. IEEE Computer, 2005.
• Linux man-pages: man 1 virsh, man 8 virt-install, man 1 qemu-system-x86_64, man 8 qemu-img, man 8 libvirtd, man 1 virt-manager, man 1 virt-clone, man 1 virt-sysprep, man 1 guestfish, man 1 lscpu, man 1 lspci, man 1 systemd-detect-virt, man 5 libvirt, man 5 qemu, man 5 qcow2.