COMP 3150Operating Systems

Module 5 · Unit 13 · Lesson 2 of 7

Hypervisors

Lesson 13.1 presented the virtual machine and the virtualization's concept. This lesson presents the hypervisor: the software that creates, runs, and manages the VMs. The hypervisor is the virtualization's central's component, and its design determines the VMs' performance, their isolation, their compatibility, and their manageability. The lesson presents the hypervisor's terminology and roles, its types (the bare-metal's, the hosted's, the hybrid's), its architectures (the monolithic's, the microkernel's, the hybrid's), the major hypervisors (the KVM, the Xen, the VMware ESXi, the Microsoft Hyper-V, the QEMU, the Firecracker, the bhyve), its components (the vCPU's scheduler, the memory's manager, the I/O's manager, the device's model), the I/O's architecture (the vhost's and the vhost-user's frameworks, the virtio's backends), the management's interfaces, the security's considerations, the live migration's mechanisms, the nested virtualization, and the microVMs.

01 Introduction

Lesson 13.1 presented the virtual machine and the virtualization's benefits (the consolidation, the isolation, the portability, the testing, the legacy's support, the high's availability) and mechanisms (the trap-and-emulate, the binary translation, the hardware's assistance, the CPU's, the memory's, and the I/O's virtualization). This lesson presents the hypervisor: the software that creates, runs, and manages the VMs. The hypervisor is the virtualization's central's component, and its design determines the VMs' performance, their isolation, their compatibility, and their manageability.

The hypervisor (also called the virtual machine monitor (VMM), the virtualizer, or the virtual machine manager) is the software layer that runs between the physical's hardware and the VMs. The hypervisor's roles:

  • The virtual hardware's emulation. The hypervisor presents the guest's operating system with the virtual hardware (the virtual CPUs, the virtual memory, the virtual devices), and it emulates the hardware's behaviors (the instructions' effects, the interrupts' delivery, the devices' protocols) that the guest's operating system expects.
  • The resources' multiplexing. The hypervisor allocates the physical's resources (the CPUs' time, the memory's space, the storage's bandwidth, the network's bandwidth) among the VMs, and it enforces the resources' isolation (the VMs' resource's usage is the contained, and the VMs cannot interfere with one another's resources).
  • The VM's lifecycle's management. The hypervisor creates the VMs (the configuration, the resources' allocation), it starts and stops them, it saves and restores their state, it migrates them between the hosts, and it destroys them (the resources' release).
  • The VMs' isolation. The hypervisor enforces the VMs' isolation: the VMs' address spaces are the separate (the memory's virtualization), the VMs' privileged's operations are the mediated (the trap-and-emulate or the hardware's assistance), and the VMs' devices are the virtualized (the emulated's devices, the virtio, the passthrough). The observation: the isolation is the hypervisor's security's foundation, and it is the reason the VMs' compromises do not spread to the other VMs or to the host.
  • The management's interfaces. The hypervisor exposes the management's interfaces (the APIs, the CLIs, the web's UIs) that the administrators and the orchestration's platforms (the OpenStack, the vCenter, the cloud's services) use to manage the VMs.

The hypervisor's position is the virtualization's architecture's central's design's decision: the hypervisor can be the bare-metal's (the type 1: the hypervisor runs directly on the hardware, and the guest's operating systems run on the hypervisor: the VMware's ESXi, the Xen, the Microsoft's Hyper-V's hypervisor), the hosted's (the type 2: the hypervisor runs as the application on the host's operating system, and the guest's operating systems run on the hypervisor: the VMware's Workstation and Fusion, the Oracle's VirtualBox, the QEMU's user's mode), or the hybrid's (the type 0: the hypervisor and the kernel are the integrated: the KVM, the Linux's kernel's virtualization's module). The position's choice affects the hypervisor's performance (the bare-metal's is typically the fastest), its functionality (the hosted's can reuse the host's operating system's drivers and services), and its management (the hosted's is the easier to install and the maintain).

This lesson presents the hypervisors. It defines the hypervisor and the terminology (the hypervisor versus the VMM). It presents the hypervisor's types (the type 1, the type 2, the type 0) and its architectures (the monolithic's, the microkernel's, the hybrid's). It presents the major hypervisors (the KVM, the Xen, the VMware ESXi, the Microsoft Hyper-V, the QEMU, the Firecracker, the bhyve) and their designs. It presents the hypervisor's components (the vCPU's scheduler, the memory's manager, the I/O's manager, the device's model, the timers, the interrupt controller). It presents the I/O's architecture (the QEMU's device model, the vhost's and the vhost-user's frameworks, the virtio's backends, the IOMMU and the VFIO). It presents the management's interfaces (the libvirt, the vSphere's API, the WMI, the XenAPI, the OpenStack) and the hypervisor's security (the attack surface, the escapes, the confidential computing). It presents the live migration's mechanisms, the nested virtualization, the microVMs, and the hypervisor's evolution. It closes with the hypervisor's design's trade-offs and the lesson's synthesis.

Prerequisites

This lesson builds on Lesson 13.1 and the earlier lessons. The virtual machine's concept, the virtualization's benefits, the hypervisor's basic's role, the types of virtualization (the full, the paravirtualization, the hardware's assistance), the trap-and-emulate's technique, the x86's virtualization's challenge, and the CPU's, the memory's, and the I/O's virtualization's mechanisms are assumed (Lesson 13.1). The process and the thread's concepts (Unit 3), the CPU's scheduling (Unit 5), the memory's management and the virtual memory (Units 8 and 9), the I/O and the storage (Unit 11), and the security's principles (Lesson 12.1) are assumed. Familiarity with the Linux command line from Lesson 2.2 is helpful, since the interactive activity uses the virsh, the qemu-system-x86_64, the ps, the lsof, the lsmod, and the systemd-detect-virt commands.

02 Learning objectives

By the end of this lesson you will be able to

  • Define the hypervisor and distinguish it from the VMM, and describe the hypervisor's roles (the virtual hardware's emulation, the resources' multiplexing, the lifecycle's management, the isolation, the management's interfaces).
  • Compare the hypervisor's types (the type 1 bare-metal, the type 2 hosted, the type 0 hybrid) and its architectures (the monolithic, the microkernel, the hybrid), and explain each design's advantages and disadvantages.
  • Describe the major hypervisors (the KVM, the Xen, the VMware ESXi, the Microsoft Hyper-V, the QEMU, the Firecracker, the bhyve) and each one's design's rationale.
  • Describe the hypervisor's components (the vCPU's scheduler, the memory's manager, the I/O's manager, the device's model, the timers, the interrupt controller) and the I/O's architecture (the vhost, the vhost-user, the virtio's backends, the IOMMU, the VFIO).
  • Describe the management's interfaces (the libvirt, the vSphere's API, the WMI, the XenAPI, the OpenStack) and the hypervisor's security (the attack surface, the escapes, the confidential computing).
  • Explain the live migration's mechanisms, the nested virtualization, and the microVMs, and evaluate a hypervisor's configuration against the lesson's criteria.

This lesson contributes to U13.2 and U13.3, and to course learning outcomes CLO1, CLO7, and CLO8.

03 Required reading

  • Operating System Concepts — Chapter 18, Virtual Machines. The standard treatment of the hypervisor's role, the hypervisor's types, and the virtualization's architectures.
  • OSTEP — the Virtualization chapters, particularly the discussion of the VMM and the containers. OSTEP's treatment of the hypervisor's components is the clearest available account, and it is the basis for the worked example in this lesson.
  • Tanenbaum & Bos — Modern Operating Systems, Chapter 7, Virtualization and the Cloud. A complementary treatment that develops the hypervisor's designs from first principles.
  • Smith, J. E., & Nair, R. — The Architecture of Virtual Machines. IEEE Computer, 2005. The comprehensive survey of the hypervisors' architectures and the VMMs' designs.
  • Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., & Warfield, A. — Xen and the Art of Virtualization. SOSP, 2003. The paper that introduced the Xen's microkernel's architecture and the paravirtualization.
  • Kivity, A., Kamay, Y., Laor, D., Lublin, U., & Liguori, A. — KVM: the Linux Virtual Machine Monitor. Ottawa Linux Symposium, 2007. The paper that describes the KVM's design (the hybrid's architecture: the Linux's kernel as the hypervisor).
  • Waldspurger, C. A. — Memory Resource Management in VMware ESX Server. OSDI, 2002. The paper that describes the VMware's memory's management (the ballooning, the page's sharing, the compression).
  • Habib, I. — Virtualization with KVM. Packt, 2008. The practical's reference for the KVM's deployment and the libvirt's management.
  • Chirammal, H. D., Mukhedkar, P., & Vettathu, A. — Mastering KVM Virtualization. Packt, 2016. The comprehensive's reference for the KVM's architecture and the virtio's and the vhost's frameworks.
  • Agache, A., Brooker, M., Iordache, A., Liguori, A., Neugebauer, R., Piwonka, P., & Popa, D. M. — Firecracker: Lightweight Virtualization for Serverless Applications. NSDI, 2020. The paper that describes the Firecracker's microVM's design.
  • The libvirt's documentation — The libvirt's API's reference, the virsh's manual, and the domain's XML's schema. The authoritative references for the libvirt's management's interfaces.
  • The Linux man pages: man 1 virsh, man 8 virt-install, man 1 qemu-system-x86_64, man 8 qemu-img, man 8 libvirtd, man 5 libvirt, man 5 qemu, man 5 qcow2, man 8 vhost (or the vhost's documentation), man 8 vhost-user (or the vhost-user's documentation), man 8 vfio, man 8 iommu, man 1 lscpu, man 1 systemd-detect-virt, man 1 dmesg, man 5 proc (for the /proc/interrupts and the /proc/cpuinfo's virtualization's information), man 1 ps, man 1 lsof, man 8 lsmod, man 1 taskset, man 1 numactl, man 1 perf, man 1 bpftrace (if available).

Reading strategy

Read the Silberschatz chapter first for the formal's context: the hypervisor's role, the hypervisor's types, and the virtualization's architectures. Then read the Smith and Nair's paper for the comprehensive's survey: the hypervisors' architectures, the VMMs' designs, and the virtualization's techniques. The Xen's and the KVM's papers are the modern's hypervisors' primary's sources, and they are worth reading for the microkernel's and the hybrid's designs: the Xen's paper presents the paravirtualization and the microkernel's architecture (the hypervisor's minimal's design, the drivers' domains), and the KVM's paper presents the hybrid's design (the Linux's kernel as the hypervisor, the QEMU as the device's model). The Waldspurger's paper is the VMware's memory's management's source, and it is worth reading for the ballooning, the page's sharing, and the compression's designs. The Firecracker's paper is the microVM's source, and it is worth reading for the lightweight's virtualization's design (the serverless's computing's requirements). The libvirt's documentation is the management's interfaces' authoritative's reference. Finally, the Linux man pages are the practical's tools: the virsh manages the VMs, the ps and the lsof inspect the hypervisor's processes and their resources, the lsmod shows the loaded's hypervisor's modules, and the systemd-detect-virt detects the virtualization's type. Pay particular attention to the distinction between the hypervisor (the software layer that runs the VMs and multiplexes the resources) and the VMM (the component that emulates the virtual hardware: the CPU's, the memory's, the devices' emulation). The hypervisor includes the VMM, but it also includes the scheduler, the memory's manager, the I/O's manager, and the management's interfaces. The observation is the reason the modern's hypervisors (the KVM, the Xen, the ESXi) are the multiple's components' systems, not the single's VMMs, and the reason the terms are often used interchangeably in the informal's usage.

04 Core content

4.1 What a hypervisor is

The hypervisor (the virtual machine monitor, the VMM) is the software layer that runs between the physical's hardware and the VMs. Its roles:

  • The virtual hardware's emulation. The hypervisor presents the guest's operating system with the virtual hardware and emulates the hardware's behaviors. The emulation's scope: the CPU's (the instructions' effects, the privileged's instructions' traps), the memory's (the address's translation, the protection), the devices' (the registers, the commands, the interrupts), and the platform's (the timers, the APIC, the firmware).
  • The resources' multiplexing. The hypervisor allocates the physical's resources among the VMs: the vCPUs' scheduling (the physical's CPUs' time's allocation), the memory's management (the physical's memory's allocation, the over-commitment's management), the storage's I/O (the bandwidth's allocation, the queues' management), and the network's I/O (the bandwidth's allocation, the virtual switches).
  • The lifecycle's management. The hypervisor creates the VMs (the definition, the resources' allocation, the devices' configuration), starts and stops them, pauses and resumes them, saves and restores their state, migrates them between the hosts, and destroys them (the resources' release).
  • The isolation. The hypervisor enforces the VMs' isolation: the memory's isolation (the separate's address spaces, the page's tables), the CPU's isolation (the vCPUs' separate's contexts), the devices' isolation (the virtual devices, the IOMMU's protection for the passthrough's devices), and the resource's isolation (the quotas, the limits).
  • The management's interfaces. The hypervisor exposes the management's interfaces: the APIs (the libvirt's, the vSphere's, the Hyper-V's WMI), the CLIs (the virsh, the vim-cmd, the xe), the web's UIs (the vCenter, the oVirt, the Proxmox), and the orchestration's platforms' integrations (the OpenStack, the Kubernetes's KubeVirt, the cloud's services).

The hypervisor's terminology: the terms hypervisor and VMM are often used interchangeably, but the precise's distinction is the following: the hypervisor is the complete's system (the VMM plus the scheduler, the memory's manager, the I/O's manager, the management's interfaces, and the security's mechanisms), and the VMM is the component that emulates the virtual hardware (the CPU's, the memory's, the devices' emulation). The observation: the classical's literature (the Goldberg's survey, the Popek and Goldberg's paper) uses the VMM's term, and the modern's usage prefers the hypervisor's term, and the two refer to the same software in the modern's context. The lesson uses the hypervisor as the primary's term and the VMM when the emulation's component is the focus.

4.2 The hypervisor's types

The hypervisor's types (the position in the software's stack):

The type 1 (the bare-metal's hypervisor)
The hypervisor runs directly on the physical's hardware, and the guest's operating systems run on the hypervisor. The type 1's design: the hypervisor is the operating system (it manages the hardware's resources, the CPU's scheduling, the memory, the devices), and the guest's operating systems are the hypervisor's domains (the isolated's environments). The type 1's advantages: the performance (the hypervisor's direct's access to the hardware: the no host's operating system's overhead), the security (the minimal's trusted's computing's base: the hypervisor's code is the small and the reviewable), and the reliability (the hypervisor's dedicated's design). The type 1's disadvantages: the hardware's compatibility (the hypervisor's drivers are the limited: the specific's hardware's support), and the management (the hypervisor's installation and the maintenance are the specialized's skills). The type 1's examples: the VMware's ESXi, the Xen's, the Microsoft's Hyper-V's (the hypervisor's component), the KVM's (the hybrid's: the Linux's kernel acts as the type 1's hypervisor's foundation), and the IBM's z/VM.
The type 2 (the hosted's hypervisor)
The hypervisor runs as an application on the host's operating system, and the guest's operating systems run on the hypervisor. The type 2's design: the host's operating system manages the hardware's resources, and the hypervisor is the user-space's application that uses the host's operating system's services (the file systems, the network's stack, the devices' drivers). The type 2's advantages: the compatibility (the host's operating system's drivers: the broad's hardware's support), the convenience (the hypervisor's installation is the application's installation), and the functionality (the host's operating system's services: the snapshots, the shared's folders, the USB's passthrough). The type 2's disadvantages: the performance (the host's operating system's overhead: the system calls, the context switches, the I/O's path's length), and the security (the host's operating system's attack surface: the hypervisor's compromise is the host's compromise). The type 2's examples: the VMware's Workstation and Fusion, the Oracle's VirtualBox, the QEMU's user's mode (without the KVM), the Parallels's Desktop, and the Microsoft's Virtual PC.
The type 0 (the hybrid's hypervisor)
The hypervisor and the kernel are the integrated: the kernel's code includes the hypervisor's functionality, and the guest's operating systems run on the kernel's hypervisor's module. The type 0's design: the kernel (the Linux's) is the host's operating system (it manages the hardware's resources for the host's processes and the VMs), and the hypervisor's module (the KVM's kvm.ko and the kvm_intel.ko's or the kvm_amd.ko's) is the kernel's component that provides the virtualization's interfaces (the /dev/kvm's device, the ioctl's interfaces, the VMX's and the SVM's operations). The type 0's advantages: the performance (the kernel's direct's access to the hardware: the no user-space's hypervisor's overhead), the integration (the kernel's services: the scheduler, the memory's management, the drivers, the security's modules: the SELinux's, the cgroups'), and the management (the standard's Linux's tools: the systemd, the virsh, the perf, the bpftrace). The type 0's disadvantages: the kernel's attack surface (the kernel's vulnerabilities are the hypervisor's vulnerabilities: the kernel's bugs can be the VM's escapes), and the complexity (the kernel's integration: the KVM's code is the part of the Linux's kernel, and the QEMU's device model is the separate's component). The type 0's examples: the KVM (the Linux's), the bhyve (the FreeBSD's), and the Microsoft's Hyper-V's (the hybrid's aspects: the Hyper-V's hypervisor's runs as the Windows's component).

The types' comparison:

The hypervisor's types' comparison
Type The position The performance The compatibility The examples
Type 1 (the bare-metal's) The hypervisor runs on the hardware; the guests run on the hypervisor The highest (the direct's access) The limited (the hypervisor's drivers) The VMware ESXi, the Xen, the Hyper-V's hypervisor, the z/VM
Type 2 (the hosted's) The hypervisor runs on the host's OS; the guests run on the hypervisor The moderate (the host's OS's overhead) The broad (the host's OS's drivers) The VMware Workstation and Fusion, the VirtualBox, the QEMU's user's mode
Type 0 (the hybrid's) The hypervisor and the kernel are the integrated The high (the kernel's integration) The broad (the kernel's drivers) The KVM, the bhyve, the Hyper-V's aspects

4.3 The hypervisor's architectures

The hypervisor's architectures (the internal's design):

The monolithic's architecture
The hypervisor is the single, the large's software's system that includes the VMM, the scheduler, the memory's manager, the I/O's manager, the device's drivers, and the management's interfaces. The monolithic's advantages: the performance (the components' direct's communication: the no IPC's overhead), and the simplicity (the single's codebase: the easier's development). The monolithic's disadvantages: the large's trusted's computing's base (the hypervisor's code is the large: the many lines of the code, the many bugs), and the reliability (the driver's bug can crash the whole's hypervisor). The monolithic's examples: the VMware's ESXi (the monolithic's with the modules), the KVM's aspects (the Linux's kernel's monolithic's design), and the Microsoft's Hyper-V's (the microkernel's aspects: the child's partitions).
The microkernel's architecture
The hypervisor's core is the minimal (the VMM, the scheduler, the memory's manager, the IPC's mechanisms), and the other's components (the device's drivers, the network's stack, the storage's stack, the management's services) run as the separate's components (the domains or the service's VMs) outside the hypervisor's core. The microkernel's advantages: the small's trusted's computing's base (the hypervisor's core is the small: the easier's verification), the reliability (the driver's bug crashes the driver's domain, not the hypervisor), and the security (the components' isolation). The microkernel's disadvantages: the performance (the components' IPC's overhead: the communication between the driver's domain and the hypervisor), and the complexity (the components' design and the communication's protocols). The microkernel's examples: the Xen (the classical's microkernel: the hypervisor's core is the small, and the drivers run in the dom0's domain), the seL4's verified's microkernel, and the QNX's.
The hybrid's architecture
The hypervisor combines the monolithic's and the microkernel's designs: the hypervisor's core includes the performance's critical's components (the VMM, the scheduler, the memory's manager, the I/O's fast's paths), and the other's components (the device's drivers, the management's services) run as the separate's components or the kernel's modules. The hybrid's advantages: the performance (the critical's components are the integrated) and the modularity (the optional's components are the separate). The hybrid's disadvantages: the complexity (the design's boundaries: the which components are the core's and which are the modular's). The hybrid's examples: the KVM (the Linux's kernel's core with the QEMU's user-space's device model), the Microsoft's Hyper-V (the hypervisor's core with the parent's partition's services), and the VMware's ESXi (the monolithic's with the modular's drivers).

The architectures' design's observations:

  • The trusted's computing's base (TCB)'s trade-off. The architecture's choice determines the TCB's size: the microkernel's TCB is the small (the hypervisor's core: the tens of the thousands of the lines), and the monolithic's TCB is the large (the hypervisor plus the drivers: the millions of the lines). The observation: the small's TCB is the easier to verify (the formal's methods, the code's audit), and the microkernel's design (the Xen, the seL4) is the high-security's environments' preference.
  • The performance's trade-off. The architecture's choice affects the performance: the monolithic's and the hybrid's designs have the faster's paths (the direct's calls), and the microkernel's design has the IPC's overhead (the components' communication). The modern's systems' benchmarks show the microkernel's overhead is the 5–15% for the I/O's intensive's workloads, and the hybrid's designs (the KVM's) have the near-native's performance.
  • The management's trade-off. The architecture's choice affects the management: the monolithic's designs have the single's management's interface, and the microkernel's designs have the components' separate's management (the driver's domains, the service's VMs). The observation: the microkernel's designs are the more complex to manage, and the management's platforms (the XenServer, the Xen Orchestra) provide the unified's interfaces.

4.4 The major hypervisors

The major hypervisors and their designs:

The KVM (the Kernel-based Virtual Machine)
The KVM is the Linux's kernel's virtualization's module (the Linux 2.6.20's integration in 2007), and it is the type 0's (the hybrid's) hypervisor. The KVM's design: the kernel's modules (the kvm.ko and the vendor's kvm_intel.ko or the kvm_amd.ko) provide the virtualization's interfaces (the /dev/kvm's device, the KVM_RUN's ioctl, the memory's slots, the vCPUs' file descriptors), and the user-space's program (the QEMU, the crosvm, the rust-vmm's based's VMMs, the Firecracker) provides the device's model (the emulated's devices, the virtio's backends) and the VM's lifecycle's management. The KVM's advantages: the performance (the kernel's direct's access, the hardware's assistance's use), the integration (the Linux's kernel's services: the scheduler, the memory's management, the drivers, the security's modules), and the flexibility (the user-space's VMMs: the different's designs: the QEMU's general's purpose, the Firecracker's microVM, the crosvm's Chromebook's, the cloud's hypervisors). The KVM's disadvantages: the kernel's attack surface (the kernel's vulnerabilities are the KVM's vulnerabilities), and the management's complexity (the two's components: the kernel's module and the user-space's VMM).
The Xen (the Xen Project's hypervisor)
The Xen is the microkernel's hypervisor (the Cambridge's Computer Laboratory's project, the 2003's release), and it is the type 1's (the bare-metal's) hypervisor. The Xen's design: the hypervisor's core (the VMM, the scheduler, the memory's manager, the minimal's devices' management), the domains (the isolated's environments: the dom0's privileged's domain for the management and the drivers, the domU's unprivileged's domains for the guests), and the paravirtualization's or the HVM's (the hardware's virtualization's) modes for the guests. The Xen's advantages: the security (the small's TCB: the hypervisor's core is the small and the verified), the isolation (the driver's domain's separation), and the scalability (the high's consolidation: the many VMs per host). The Xen's disadvantages: the performance (the IPC's overhead for the I/O's paths: the driver's domain's communication), and the management's complexity (the multiple's components: the hypervisor, the dom0, the toolstack).
The VMware ESXi (the VMware's hypervisor)
The ESXi is the VMware's type 1's hypervisor (the ESX's successor, the 2008's release), and it is the enterprise's virtualization's standard. The ESXi's design: the VMkernel (the proprietary's kernel: the scheduler, the memory's manager, the storage's stack, the network's stack, the drivers), the VMM (the per-VM's process: the vCPUs' execution, the memory's management, the devices' emulation), and the vCenter (the management's platform: the central's management, the advanced's features: the vMotion, the DRS, the HA). The ESXi's advantages: the performance (the mature's design, the optimized's paths), the features (the vMotion's live's migration, the DRS's load's balancing, the HA's failover, the FT's fault's tolerance), and the ecosystem (the vSphere's management's platform, the third-party's integrations). The ESXi's disadvantages: the cost (the licenses are the expensive), the vendor's lock-in (the proprietary's formats and the APIs), and the closed's design (the source's code is the closed).
The Microsoft Hyper-V
The Hyper-V is the Microsoft's hypervisor (the Windows Server 2008's introduction, the Windows 8's and the Windows 10's client's versions), and it is the type 1's (the bare-metal's) hypervisor with the type 0's aspects (the integration with the Windows's operating system). The Hyper-V's design: the hypervisor (the minimal's core: the VMM, the scheduler, the memory's manager, the IPC's mechanisms), the parent's partition (the management's operating system: the Windows's with the virtualization's stack), and the child's partitions (the guest's VMs). The Hyper-V's advantages: the integration (the Windows's ecosystem: the Active Directory, the System Center, the PowerShell's management), the features (the live's migration, the replicas, the checkpoints, the shielded's VMs), and the licensing (the included with the Windows Server). The Hyper-V's disadvantages: the Windows's dependency (the parent's partition is the Windows: the Linux's support is the second-class), and the performance (the Windows's overhead in the parent's partition).
The QEMU (the Quick Emulator)
The QEMU is the user-space's emulator and the VMM's component (the Fabrice Bellard's project, the 2003's release). The QEMU's design: the QEMU is the user-space's program that provides the device's model (the emulated's devices: the IDE's, the SCSI's, the network's, the USB's, the VGA's, the sound's, and the virtio's paravirtualized's devices) and the VM's lifecycle's management (the vCPUs' threads, the memory's allocation, the devices' configuration, the monitor's console). The QEMU's modes: the TCT (the Tiny Code Generator: the dynamic's binary translation: the QEMU's cross-architecture's emulation: the ARM's code on the x86's host), the KVM's mode (the QEMU uses the KVM's kernel's module for the CPU's and the memory's virtualization, and the QEMU provides the device's model: the modern's standard), and the user's mode (the QEMU's Linux's user-space's emulation: the single's process's execution, the qemu-arm's commands). The QEMU's advantages: the flexibility (the many's emulated's devices and the many's architectures), the openness (the open-source, the broad's community), and the integration (the libvirt's management, the KVM's hardware's assistance). The QEMU's disadvantages: the performance (the emulation's overhead: the TCG's translation, the device's emulation), and the security (the QEMU's user-space's attack surface: the device's model's vulnerabilities: the VENOM's).
The Firecracker (the microVM's hypervisor)
The Firecracker is the AWS's microVM's hypervisor (the 2018's release, the open-source in 2020), designed for the serverless's and the container's workloads (the AWS's Lambda and the Fargate). The Firecracker's design: the minimal's VMM (the Rust's implementation: the vCPUs, the memory, the minimal's devices: the virtio-net, the virtio-block, the virtio-vsock, the serial's console), the KVM's kernel's module for the CPU's and the memory's virtualization, and the jailer (the sandboxing's component: the chroot, the namespaces, the seccomp's filters, the cgroups). The Firecracker's advantages: the fast's startup (the 125 ms's boot's time), the low's memory's overhead (the ~5 MB per microVM), the strong's isolation (the VM's boundary: the stronger than the containers), and the minimal's attack surface (the small's codebase: the ~50,000's lines of the Rust's code). The Firecracker's disadvantages: the limited's devices (the microVM's design: the minimal's devices), and the specific's use's case (the serverless's and the container's workloads: the not the general's purpose).
The bhyve (the FreeBSD's hypervisor)
The bhyve is the FreeBSD's hypervisor (the FreeBSD 10's introduction, the 2014's release), and it is the type 2's (the hosted's) hypervisor with the type 1's aspects (the kernel's module). The bhyve's design: the kernel's module (the VMM, the vCPUs, the memory, the devices' access), the user-space's bhyve's command (the VM's creation and management), and the bhyvectl's and the vmrun's tools. The bhyve's advantages: the FreeBSD's integration (the ZFS's storage, the jails' isolation, the pf's firewall), the performance (the hardware's assistance's use), and the licensing (the BSD's: the permissive). The bhyve's disadvantages: the FreeBSD's dependency (the not the Linux's: the different's ecosystem), and the ecosystem (the fewer's tools and the integrations).

The hypervisors' comparison:

The major hypervisors' comparison
Hypervisor The type The architecture The platform The typical use
The KVM The type 0 (the hybrid's) The hybrid (the kernel's module + the user-space's VMM) The Linux The cloud's providers (the OpenStack, the GCP), the enterprise's Linux's servers
The Xen The type 1 (the bare-metal's) The microkernel The Linux's dom0, the NetBSD, the Solaris The security's environments (the Qubes OS, the AWS's early's infrastructure)
The VMware ESXi The type 1 (the bare-metal's) The monolithic with the modules The VMware's VMkernel (the proprietary) The enterprise's virtualization (the vSphere, the private's clouds)
The Microsoft Hyper-V The type 1 (the bare-metal's) The microkernel's aspects (the parent's partition) The Windows Server, the Windows 10/11 The Windows's environments, the Microsoft's clouds (the Azure)
The QEMU The type 2 (the user-space's VMM) The device's model The many (the Linux, the Windows, the macOS) The development, the testing, the KVM's device's model
The Firecracker The type 0 (the KVM's based) The minimal's VMM (the Rust) The Linux (the KVM) The serverless's and the container's workloads (the AWS's Lambda, the Fargate)
The bhyve The type 2 (the kernel's module) The hybrid The FreeBSD The FreeBSD's environments (the jails, the ZFS)
The hypervisor's types and the hypervisor's components The diagram has two parts. The top part compares the hypervisor's three types: the type 1 (the bare-metal's hypervisor: the hypervisor runs on the hardware and the guests on the hypervisor), the type 2 (the hosted's hypervisor: the host's operating system runs on the hardware, the hypervisor on the host's operating system, and the guests on the hypervisor), and the type 0 (the hybrid's hypervisor: the kernel includes the hypervisor's module, and the guests and the host's processes share the kernel). The bottom part shows the hypervisor's components: the VMM (the CPU, the memory, the devices' emulation), the vCPU's scheduler, the memory's manager, the I/O's manager (the device's model, the virtio, the vhost, the passthrough), the management's interfaces (the libvirt, the APIs, the CLIs), and the security's mechanisms (the isolation, the IOMMU, the confidential computing). THE HYPERVISOR'S TYPES Type 1 — the bare-metal's The guest's VMs The hypervisor (the bare-metal's) the VMM + the scheduler + the memory + the I/O + the drivers The physical hardware The highest performance; the limited drivers The VMware ESXi, the Xen, the Hyper-V's hypervisor Type 2 — the hosted's The guest's VMs The hypervisor (the application) The host's operating system The physical hardware The broad's drivers; the moderate's performance Type 0 — the hybrid's The guest's VMs The kernel + the hypervisor's module (the KVM's kvm.ko, the Linux's scheduler and memory) The physical hardware The integration's benefits; the kernel's attack surface The KVM, the bhyve THE HYPERVISOR'S COMPONENTS The VMM • The CPU's emulation • The memory's virtualization • The devices' emulation • The traps and the emulation • The guest's context's management The vCPU's scheduler • The physical's CPUs' time • The vCPUs' multiplexing • The co-scheduling (the SMP) • The over-commitment • The affinities, the shares The memory's manager • The GPA → HPA mapping • The EPT / the NPT • The shadow's page tables • The ballooning, the KSM • The over-commitment The I/O's manager • The device's model (the QEMU) • The virtio's backends • The vhost / the vhost-user • The passthrough (the VFIO, the SR-IOV) • The virtual switches The management's interfaces • The libvirt (the KVM's, the Xen's, the QEMU's, the bhyve's unified's API) • The vSphere's API (the VMware's), the WMI (the Hyper-V's), the XenAPI • The CLIs (the virsh, the xe, the vim-cmd), the web's UIs, the OpenStack The security's mechanisms • The VMs' isolation (the memory, the CPU, the devices) • The IOMMU (the VT-d, the AMD-Vi) for the passthrough's safety • The confidential computing (the SEV, the SGX, the TDX) The hypervisor's architecture's choice (the monolithic, the microkernel, the hybrid) determines the TCB's size and the components' placement The architectures' comparison The monolithic: the large's TCB, the fast's paths (the ESXi, the Hyper-V) The microkernel: the small's TCB, the IPC's overhead (the Xen, the seL4) The hybrid: the balanced's design, the kernel's integration (the KVM, the bhyve)
Figure 13.2 — The hypervisor's types and the hypervisor's components. The top diagram compares the hypervisor's three types: the type 1 (the bare-metal's: the hypervisor runs on the hardware and the guests on the hypervisor), the type 2 (the hosted's: the host's operating system runs on the hardware, the hypervisor on the host's operating system, and the guests on the hypervisor), and the type 0 (the hybrid's: the kernel includes the hypervisor's module, and the guests and the host's processes share the kernel). The bottom diagram shows the hypervisor's components: the VMM (the CPU, the memory, the devices' emulation), the vCPU's scheduler, the memory's manager, the I/O's manager (the device's model, the virtio, the vhost, the passthrough), the management's interfaces (the libvirt, the APIs, the CLIs), and the security's mechanisms (the isolation, the IOMMU, the confidential computing). The hypervisor's architecture's choice (the monolithic, the microkernel, the hybrid) determines the TCB's size and the components' placement.

4.5 The hypervisor's components

The hypervisor's components (the internal's design):

  • The VMM (the virtual machine monitor). The component that emulates the virtual hardware: the CPU's emulation (the traps and the emulation, the hardware's assistance's use, the guest's context's management), the memory's virtualization (the address's translation, the EPT's or the NPT's management, the shadow's page's tables), and the devices' emulation (the emulated's devices, the virtio's backends, the passthrough's devices). The VMM's design: the per-VM's process (the QEMU's, the Firecracker's) or the kernel's module (the KVM's) or the per-VM's component (the ESXi's VMX's, the Xen's domU's).
  • The vCPU's scheduler. The component that allocates the physical's CPUs' time among the vCPUs: the scheduling's policies (the Linux's CFS for the KVM, the credit's scheduler for the Xen, the ESXi's proportional's shares), the co-scheduling (the SMP's guests' vCPUs), the vCPUs' migration (the between the physical's CPUs), the over-commitment (the more vCPUs than the physical's CPUs), and the latency's considerations (the vCPUs' wake-up's latency, the I/O's completion's latency).
  • The memory's manager. The component that manages the physical's memory's allocation among the VMs: the GPA's to the HPA's mapping (the EPT's or the NPT's tables, the shadow's page's tables), the memory's over-commitment (the ballooning, the page's sharing (the KSM), the compression, the swapping), the memory's hot-plug, and the memory's bandwidth's management (the NUMA's awareness, the memory's tiers).
  • The I/O's manager. The component that manages the VMs' I/O: the device's model (the emulated's devices), the virtio's backends (the QEMU's user-space's backends, the vhost's kernel's backends, the vhost-user's user-space's daemons), the passthrough's devices (the VFIO, the SR-IOV), the virtual switches (the Linux's bridge, the Open vSwitch), and the storage's stack (the QCOW2's images, the shared's storage, the distributed's storage).
  • The device's model. The component that emulates the virtual devices: the QEMU's device model (the IDE's, the SCSI's, the SATA's, the NVMe's, the network's (the e1000, the virtio-net, the vmxnet3), the USB's, the VGA's, the sound's, the serial's, the RTC's, the PCI's), the virtio's devices (the virtio-net, the virtio-blk, the virtio-scsi, the virtio-gpu, the virtio-fs, the virtio-vsock), and the platform's devices (the APIC, the timers, the HPET).
  • The timers and the interrupt's controller. The components that emulate the platform's timing and the interrupts: the timers (the PIT, the HPET, the local APIC's timer, the TSC), the interrupt's controller (the PIC, the IOAPIC, the MSI's and the MSI-X's emulation, the posted's interrupts), and the clock's sources (the kvm-clock's paravirtualized's clock, the TSC's scaling).

4.6 The I/O's architecture

The I/O's architecture is the hypervisor's performance's critical's component, and its design's evolution reflects the virtualization's performance's requirements:

  • The QEMU's user-space's device model. The classical's design: the QEMU's user-space's process emulates the devices, and the guest's I/O's requests are the trapped (the VMEXIT's) and the handled in the user-space (the QEMU's device's emulation). The design's advantages: the flexibility (the many's emulated's devices), and the isolation (the device's model's bugs are the QEMU's process's bugs: the contained). The design's disadvantages: the performance (the traps and the user-space's processing's overhead: the 10–30% for the I/O's intensive's workloads).
  • The virtio's design. The paravirtualized's I/O's interfaces: the guest's drivers (the virtio's drivers) and the hypervisor's backends (the QEMU's virtio's backends) communicate through the shared's memory's rings (the virtqueues: the descriptor's tables, the available's and the used's rings) and the notifications (the kick's and the interrupt's injection). The virtio's advantages: the performance (the batched's requests, the shared's memory: the no traps for the data), and the standardization (the OASIS's specification: the interoperability). The virtio's design's components: the virtio's specification, the virtio's drivers (the Linux's, the Windows's, the BSD's), the virtio's devices (the virtio-net, the virtio-blk, the virtio-scsi, the virtio-gpu, the virtio-fs, the virtio-vsock, the virtio-input, the virtio-balloon), and the virtio's transports (the PCI's, the MMIO's, the channel's I/O's).
  • The vhost's framework. The kernel's acceleration's framework: the vhost's kernel's module (the vhost-net, the vhost-scsi, the vhost-vsock) handles the virtio's queues in the kernel, so the guest's I/O's requests are the processed without the user-space's (the QEMU's) involvement. The vhost's advantages: the performance (the kernel's processing: the no user-space's context's switches), and the simplicity (the QEMU's device model's offload). The vhost's design: the vhost's ioctl's interfaces (the VHOST_SET_OWNER, the VHOST_SET_MEM_TABLE, the VHOST_SET_VRING_*), the eventfd's notifications, and the vhost-net's data's path (the guest's virtqueue to the host's socket or the TAP's device).
  • The vhost-user's framework. The user-space's acceleration's framework: the dedicated's user-space's daemon (the OVS's vhost-user, the SPDK's vhost, the virtiofsd, the virtio-gpu's vhost-user-gpu) handles the virtio's queues in the user-space, so the I/O's processing is the user-space's (the flexibility's and the isolation's benefits). The vhost-user's advantages: the flexibility (the user-space's processing: the custom's backends), the isolation (the daemon's crash is the contained), and the performance (the shared's memory: the no copies). The vhost-user's design: the Unix's domain's socket (the control's channel), the shared's memory (the virtio's queues), the eventfd's notifications, and the VHOST_USER_*'s protocols.
  • The passthrough's design (the VFIO, the SR-IOV). The device's assignment: the hypervisor assigns the physical's device (the network's adapter, the GPU, the storage's controller) directly to the guest's VM, and the guest's driver interacts with the physical's device as if it were the exclusive's owner. The passthrough's mechanisms: the VFIO's framework (the vfio-pci's driver, the IOMMU's groups, the /dev/vfio/'s devices, the DMA's remapping via the IOMMU), the SR-IOV (the single root I/O virtualization: the physical's function (the PF) and the virtual's functions (the VFs), the VFs' assignment to the VMs), and the VFIO's no-IOMMU's mode (the unsafe's mode for the debugging). The passthrough's advantages: the near-native's performance (the no hypervisor's mediation), and the compatibility (the native's drivers). The passthrough's disadvantages: the exclusivity (the one's VM's device), the migration's difficulty (the state's transfer), and the security's considerations (the IOMMU's isolation: the mandatory's requirement).

4.7 The management's interfaces

The hypervisor's management's interfaces:

The libvirt (the unified's API)
The libvirt is the open-source's API, daemon, and toolset for managing the virtualization's platforms: the KVM, the QEMU, the Xen, the VMware ESXi, the Hyper-V, the bhyve, the LXC, the OpenVZ. The libvirt's components: the libvirtd's daemon, the libvirt's client's library (the C, the Python, the Go, the Java bindings), the virsh's CLI, the virt-manager's GUI, the virt-install's provisioning's tool, and the domain's XML's schema (the virsh dumpxml's and the virsh define's interfaces). The libvirt's advantages: the unified's interface (the same's API for the different's hypervisors), the remote's management (the TLS's and the SSH's connections), and the ecosystem (the OpenStack, the oVirt, the Proxmox, the Vagrant, the Terraform providers).
The vSphere's API (the VMware's)
The vSphere's API is the VMware's management's interface for the ESXi and the vCenter: the SOAP's and the REST's APIs, the vSphere's client's libraries (the pyvmomi for the Python, the govmomi for the Go), the govc's and the vim-cmd's CLIs, and the vCenter's web's UI. The vSphere's advantages: the enterprise's features (the vMotion, the DRS, the HA, the FT), the ecosystem (the VMware's and the third-party's tools), and the integration (the backup's, the monitoring's, the automation's platforms).
The WMI and the PowerShell (the Hyper-V's)
The Hyper-V's management's interfaces: the WMI's providers (the root\virtualization\v2's namespace), the PowerShell's cmdlets (the Hyper-V's module: the New-VM, the Start-VM, the Get-VM, the Move-VM), the vmconnect's console, and the System Center's Virtual Machine Manager (the SCVMM). The Hyper-V's advantages: the Windows's integration (the Active Directory, the Group Policy, the System Center), and the scripting (the PowerShell's automation).
The XenAPI and the XenServer
The Xen's management's interfaces: the XenAPI (the XML-RPC's interface: the xe's CLI, the XenAPI's libraries), the XenServer (the Citrix's commercial's distribution: the XenCenter's GUI, the pools, the HA, the live migration), and the Xen Orchestra (the open-source's web's UI and the API). The Xen's advantages: the pools' management (the resource's pooling), and the open-source's ecosystem (the XCP-ng's community's distribution).
The OpenStack (the cloud's platform)
The OpenStack is the open-source's cloud's platform: the Nova's compute's service (the VMs' lifecycle: the scheduling, the placement, the migration), the Neutron's network's service (the virtual networks, the routers, the security's groups), the Cinder's block's storage's service (the volumes), the Glance's image's service, the Keystone's identity's service, the Horizon's dashboard, and the Placement's service (the resources' inventory). The OpenStack's advantages: the cloud's features (the multi-tenancy, the self-service, the APIs), the ecosystem (the many's vendors and the integrations), and the openness.

4.8 The hypervisor's security

The hypervisor's security is the virtualization's security's foundation, and its considerations:

  • The attack surface. The hypervisor's attack surface: the hypervisor's code (the VMM, the device's model, the drivers), the management's interfaces (the APIs, the daemons), and the configuration (the shared's resources, the passthrough's devices, the live's migration's channels). The observation: the smaller the hypervisor's code, the smaller the attack surface, and the observation is the reason the minimal's hypervisors (the Xen's, the Firecracker's, the seL4's) are the high-security's environments' preference.
  • The VM's escapes. The VM's escape is the attacker's exploit of the hypervisor's or the guest's vulnerability to break the isolation: the classical's examples: the VENOM (the CVE-2015-3456: the QEMU's floppy's controller's buffer's overflow: the guest's code's execution on the host), the XSA's advisories (the Xen's vulnerabilities: the XSA-148's, the XSA-155's, the XSA-182's), the CVE-2019-5736 (the runc's: the container's escape, the related's virtualization's context), the CVE-2020-14364 (the QEMU's USB's controller), and the CVE-2021-3653 and the CVE-2021-3656 (the AMD's SEV's vulnerabilities). The escapes' mitigations: the hypervisor's updates (the timely's patching), the minimal's configurations (the unnecessary's devices' disabling), the hardware's assistance (the IOMMU's isolation, the NPT's and the EPT's), the monitoring (the introspection, the anomaly's detection), and the confidential computing (the SEV, the SGX, the TDX).
  • The side channels. The side channels are the information's leaks through the shared's resources: the CPU's caches (the timing's differences: the Prime+Probe, the Flush+Reload), the memory's deduplication (the KSM's page's sharing's timing), the speculative's execution (the Spectre's and the Meltdown's attacks), the power's consumption, and the electromagnetic's emissions. The side channels' mitigations: the caches' partitioning (the Intel's CAT, the AMD's QoS), the KSM's disabling (the high-security's environments), the microcode's updates, the speculative's execution's mitigations (the IBRS, the IBPB, the STIBP, the retpolines), and the constant-time's programming.
  • The confidential computing. The confidential computing protects the VMs' memory from the hypervisor and the physical's attackers: the AMD's SEV (the Secure Encrypted Virtualization: the per-VM's memory's encryption with the AMD's Secure Processor's keys), the AMD's SEV-ES and the SEV-SNP (the encrypted's state, the secure's nested's paging), the Intel's SGX (the Software's Guard Extensions: the enclaves: the protected's memory's regions), the Intel's TDX (the Trust Domain Extensions: the VMs' isolation from the hypervisor), and the ARM's CCA (the Confidential Compute Architecture: the realms). The confidential computing's advantages: the protection from the hypervisor's compromise (the cloud's provider cannot read the VM's memory), and the compliance's requirements (the regulated's workloads). The confidential computing's disadvantages: the performance (the encryption's overhead: the 5–15% for the memory's intensive's workloads), the limited's features (the restricted's I/O, the limited's live migration), and the ecosystem (the software's support: the guest's OS's support, the attestation's services).

4.9 The live migration

The live migration is the VM's transfer between the hosts with the minimal's downtime, and its mechanisms:

  • The pre-copy's phase. The hypervisor copies the VM's memory's pages to the destination's host while the VM continues to run: the initial's pass copies all the pages, and the subsequent's passes copy the pages that the VM dirtied since the previous's pass. The pre-copy's termination: the hypervisor stops the pre-copy when the dirty's pages' rate is the manageable (the smaller than the available's bandwidth) or when the iterations' limit is the reached.
  • The stop-and-copy's phase. The hypervisor pauses the VM, copies the remaining's pages (the dirty's pages and the vCPUs' registers, the devices' state), and starts the VM on the destination's host. The phase's duration is the migration's downtime (the typical: the 50–500 ms), and the observation: the downtime's magnitude is the migration's quality's metric.
  • The post-copy's phase. The alternative's technique: the hypervisor transfers the vCPUs' state and the minimal's memory to the destination, and the VM starts on the destination while the remaining's pages are the fetched on the demand (the page's faults). The post-copy's advantages: the migration's guaranteed's completion (the no pre-copy's divergence's risk), and the lower's total's transfer. The post-copy's disadvantages: the performance's degradation during the fetch's phase (the page's faults' latency), and the failure's handling (the source's host must be the available until the transfer's completion).
  • The migration's optimizations. The compression (the memory's pages' compression: the zlib, the LZ4), the deduplication (the zero's pages and the identical's pages), the multi-threading (the parallel's transfer), the RDMA (the remote's direct's memory's access: the faster's transfer for the large's VMs), and the auto-convergence (the vCPUs' throttling to reduce the dirty's pages' rate).
  • The migration's constraints. The compatible's CPUs (the features' matching: the host-model's or the host-passthrough's configurations), the shared's storage (the NFS's, the Ceph's, the iSCSI's) or the block's migration (the disks' transfer), the network's configuration (the virtual networks' consistency), and the compatibility (the hypervisors' versions, the devices' models).

4.10 The nested virtualization and the microVMs

The nested virtualization and the microVMs are the hypervisors' modern's developments:

  • The nested virtualization. The hypervisor's capability to run the VMs inside the VMs: the L0's hypervisor (the physical's) runs the L1's VM (the guest that is the hypervisor), and the L1's hypervisor runs the L2's VMs (the guest's guests). The nested virtualization's mechanisms: the hardware's assistance's extensions (the Intel's vmx's nested's support, the AMD's SVM's nested's paging), the L0's hypervisor's emulation (the L1's hypervisor's privileged's operations are the trapped and the emulated by the L0), and the performance's considerations (the multiple's translation's levels: the GVA → the GPA → the GPA (the L1's) → the HPA). The nested virtualization's uses: the development (the testing of the hypervisors), the cloud's services (the nested's VMs for the customers), and the security's research (the microkernels, the sandboxes). The nested virtualization's performance: the 10–30% overhead for the L2's VMs (the additional's translation's level).
  • The microVMs. The lightweight's VMs designed for the serverless's and the container's workloads: the minimal's device's model, the minimal's guest's kernel, the fast's startup (the tens of the milliseconds), and the low's memory's overhead (the few MBs). The microVMs' examples: the AWS's Firecracker (the firecracker's VMM: the ~50,000's lines of the Rust's code, the ~125 ms's boot's time, the ~5 MB's memory's overhead, the jailer's sandboxing), the Google's gVisor (the user-space's kernel: the application's kernel's emulation in the user-space), the Kata Containers (the containers' lightweight's VMs: the VM's isolation with the container's interfaces), and the QEMU's microvm's machine's type (the minimal's device's model). The microVMs' advantages: the strong's isolation (the VM's boundary: the stronger than the containers), the fast's startup (the serverless's requirements), and the minimal's attack surface (the small's codebase). The microVMs' disadvantages: the limited's devices (the minimal's device's model), and the specific's use's case (the serverless's and the container's workloads).

4.11 The hypervisor's evolution and the trends

The hypervisor's evolution and the trends:

  • The hardware's assistance's completeness. The CPU's vendors continue the virtualization's extensions: the memory's virtualization (the EPT's and the NPT's enhancements: the 5-level's page's tables), the I/O's virtualization (the SR-IOV's enhancements, the scalable's IOV, the Intel's VT-d's and the AMD's AMD-Vi's updates), the interrupt's virtualization (the posted's interrupts, the APICv), the security's extensions (the CET, the SGX, the SEV, the TDX), and the nested's virtualization's support.
  • The hypervisor's specialization. The general-purpose's hypervisors (the KVM, the Xen, the ESXi) are the mature, and the specializations emerge: the microVMs (the Firecracker's), the unikernels (the MirageOS's, the IncludeOS's, the Nanos's), the real-time's hypervisors (the Jailhouse's, the ACRN's, the Xen's real-time's configurations), the automotive's hypervisors (the QNX's, the AGL's), and the embedded's hypervisors (the ARM's virtualization's extensions).
  • The cloud's native's hypervisors. The cloud's providers' custom's hypervisors: the AWS's Nitro (the dedicated's hardware's cards: the network's, the storage's, the management's offloads), the Google's (the KVM's with the custom's scheduling and the security's enhancements), the Microsoft's (the Hyper-V's with the Azure's enhancements), and the Alibaba's (the Dragonfly's hypervisor). The trend: the hypervisor's functionality's offload to the dedicated's hardware (the SmartNICs, the DPUs), the security's enhancements (the confidential's computing), and the performance's optimizations (the SR-IOV's and the passthrough's devices).
  • The container's integration. The hypervisors and the containers' convergence: the Kata Containers (the containers with the VM's isolation), the gVisor (the user-space's kernel), the Firecracker (the microVMs for the containers), the KubeVirt (the VMs' management in the Kubernetes's clusters), and the virtio's containers (the libkrun's, the crun's). The observation: the VMs' and the containers' boundaries are the blurring, and the observation is the reason the modern's platforms (the Kubernetes's, the cloud's) support the both.
  • The confidential computing's adoption. The confidential computing's mechanisms (the AMD's SEV-SNP, the Intel's TDX, the ARM's CCA) and the ecosystem (the attestation's services, the confidential's containers: the enclave-cc's, the occlum's, the gramine's). The trend: the regulated's workloads (the healthcare, the finance, the government) and the multi-tenant's clouds' requirements are the confidential computing's drivers.

4.12 Common misconceptions

"The hypervisor is the same as the VMM."

The two terms are often used interchangeably, but the precise's distinction: the hypervisor is the complete's system (the VMM plus the scheduler, the memory's manager, the I/O's manager, the management's interfaces, the security's mechanisms), and the VMM is the component that emulates the virtual hardware. The observation: the classical's literature uses the VMM's term, and the modern's usage prefers the hypervisor's term, and the two refer to the same software in the modern's context.

"The type 1's hypervisor is always the faster."

The type 1's hypervisor's performance advantage is the typical, but the difference depends on the workload and the hypervisor: the modern's type 0's hypervisors (the KVM's) achieve the near-native's performance (the kernel's integration, the hardware's assistance), and the type 2's hypervisors (the VirtualBox's, the Workstation's) can be the adequate for the development and the testing's workloads. The observation: the hypervisor's type is the design's decision, and its performance depends on the whole's design (the device's model, the I/O's paths, the hardware's assistance).

"The microkernel's hypervisor is the always's slower."

The microkernel's hypervisor has the IPC's overhead (the components' communication: the driver's domain, the hypervisor's core), and the overhead is the 5–15% for the I/O's intensive's workloads. But the microkernel's design has the advantages: the small's TCB (the security's and the reliability's benefits), the driver's isolation (the driver's crash is the contained), and the observation is the reason the microkernel's designs (the Xen's, the seL4's) are the high-security's environments' preference. The choice depends on the priorities (the performance's vs. the security's).

"The QEMU is the hypervisor."

The QEMU is the emulator and the VMM's component, not the complete's hypervisor: the QEMU provides the device's model and the VM's lifecycle's management, and the KVM's kernel's module provides the CPU's and the memory's virtualization. The observation: the QEMU can run without the KVM (the user-space's emulation: the slower, the cross-architecture's), and the KVM can run with the other's VMMs (the Firecracker's, the crosvm's, the rust-vmm's based's VMMs). The two components' combination is the standard's configuration.

"The live migration is the seamless."

The live migration has the downtime (the stop-and-copy's phase: the 50–500 ms for the typical's VMs), the bandwidth's requirements (the memory's size and the dirty's pages' rate), and the constraints (the CPUs' features, the shared's storage or the block's migration, the network's configuration). The observation: the live migration is the powerful's mechanism, and its design must account for the constraints and the failure's modes (the timeouts, the bandwidth's exhaustion, the incompatible's configurations).

"The hypervisor's vulnerabilities are the rare."

The hypervisor's vulnerabilities are the recurring: the VENOM (the QEMU's floppy's controller), the Xen's XSA's advisories, the KVM's bugs, the Hyper-V's vulnerabilities, the VirtualBox's escapes. The observation: the hypervisor's security is the ongoing's discipline (the timely's patching, the minimal's configurations, the monitoring), and the observation is the reason the high-security's environments use the minimal's hypervisors and the hardware's assistance (the IOMMU's isolation, the confidential computing).

"The microVMs are the same as the containers."

The microVMs (the Firecracker's) and the containers (the Docker's, the Podman's) are the different's technologies: the microVMs use the KVM's virtualization (the VM's boundary: the separate's guest's kernel), and the containers use the namespaces and the cgroups (the shared's kernel: the Unit 12's Lesson 6). The microVMs' advantages: the strong's isolation (the VM's boundary), and the containers' advantages: the density (the hundreds per host) and the resource's usage (the MBs). The modern's platforms combine the two (the Kata Containers: the containers' interfaces with the microVMs' isolation; the Firecracker's microVMs for the AWS's Lambda and the Fargate).

"The hypervisor's choice is the permanent."

The hypervisor's choice can be the changed: the VM's images can be the converted (the qemu-img's conversions, the virt-v2v's migrations: the VMware's to the KVM's, the Hyper-V's to the KVM's), the workloads can be the migrated (the live migration, the cold migration, the backup and the restore), and the management's platforms can be the changed. The observation: the hypervisor's choice is the consequential, but it is not the permanent, and the observation is the reason the modern's virtualization's practices include the portability (the standard's images: the OVA's, the QCOW2's, the OVF's, the cloud's formats) and the management's abstraction (the libvirt's, the Terraform's).

05 Worked example

The following example inspects a KVM-based hypervisor's components on a live system, traces a VM's I/O's path through the virtio and the vhost's frameworks, and demonstrates the live migration's configuration.

Inspecting the hypervisor's components

The following commands inspect the KVM-based hypervisor's components: 

# The hypervisor's modules
lsmod | grep -E 'kvm|vfio|vhost'
ls /sys/module/kvm/ 2>/dev/null
ls /sys/module/kvm_intel/parameters/ 2>/dev/null
cat /sys/module/kvm_intel/parameters/nested 2>/dev/null

# The /dev/kvm's device
ls -la /dev/kvm
cat /proc/misc | grep kvm

# The virtualization's capabilities
virsh capabilities 2>/dev/null | head -40

# The QEMU's process (if a VM runs)
ps -ef | grep qemu
ps -eLf | grep qemu | wc -l  # the threads' count

# The QEMU's file descriptors (the devices, the disks, the network)
sudo ls -la /proc/$(pgrep -f qemu | head -1)/fd 2>/dev/null | head -20

# The vhost's kernel's threads
ps -ef | grep '\[vhost' | head -5

# The VM's vCPU's threads
ps -eLf | grep -E 'CPU [0-9]+/KVM' | head -5

# The VM's cgroup's placement
systemd-cgls /machine.slice/ 2>/dev/null | head -20

# The IOMMU's groups (if the passthrough is used)
ls /sys/kernel/iommu_groups/ 2>/dev/null | head -10

# The interrupt's distribution
grep -E 'vfio|virtio' /proc/interrupts | head -10

# The virtualization's detection
systemd-detect-virt
lscpu | grep -E 'Virtualization|Hypervisor'

The components' inspection's findings:

  • The hypervisor's modules. The kvm.ko (the KVM's core), the kvm_intel.ko (the Intel's VT-x's support: the nested's parameter for the nested virtualization), the vfio_pci and the vfio_iommu_type1 (the VFIO's framework for the passthrough's devices), and the vhost_net (the vhost's kernel's framework).
  • The QEMU's process. The QEMU's main's process (the VM's management: the devices, the monitor, the migration), the vCPUs' threads (the CPU X/KVM's threads: one per vCPU), and the I/O's threads (the virtio's processing, the vhost's threads).
  • The vhost's kernel's threads. The [vhost-<PID>]'s threads: the vhost's kernel's threads that handle the virtio's queues in the kernel (the network's and the storage's paths).
  • The VM's cgroup's placement. The VM's processes (the QEMU's, the vhost's) run in the machine.slice's cgroup (the machine-qemu\x2d1\x2ddemo\x2dvm.scope's or the libvirt-qemu-demo-vm.scope's cgroup), and the cgroup's controllers (the cpu, the memory, the io, the pids) limit the VM's resource's usage.

The I/O's path's trace

The following trace follows a guest's disk's read through the virtio-blk's and the vhost-user-blk's paths:

  1. The guest's application's read. The guest's application reads a file, and the guest's file system issues the block's read to the virtio-blk's driver.
  2. The virtio-blk's driver's request. The virtio-blk's driver writes the request's descriptor to the virtqueue (the shared's memory's ring: the descriptor's table, the available's ring), and it notifies the backend (the kick's notification: the MMIO's write or the I/O's port's write).
  3. The backend's processing. The backend (the QEMU's user-space's virtio-blk, the vhost-blk's kernel's thread, or the vhost-user-blk's user-space's daemon) reads the descriptor's table, translates the guest's buffers' addresses (the GVA to the GPA to the HPA), and submits the I/O to the physical's device (the host's file, the block's device, or the SPDK's user-space's driver).
  4. The physical's device's completion. The physical's device (the host's storage) completes the I/O, and the backend updates the virtqueue's used's ring and injects the completion's interrupt to the guest (the eventfd's notification, the irqfd's injection, the vCPU's kick).
  5. The guest's completion. The guest's virtio-blk's driver's interrupt's handler processes the completion, and the guest's file system completes the read.

The I/O's path's variants:

The backend The location The performance The use's case
The QEMU's virtio-blk The QEMU's user-space The moderate (the user-space's context's switches) The development, the compatibility
The vhost-blk The kernel's thread The high (the kernel's processing) The performance's workloads
The vhost-user-blk The user-space's daemon (the SPDK, the virtiofsd) The high (the shared's memory, the polling) The high-performance's workloads (the SPDK's user-space's drivers)
The VFIO's passthrough The guest's kernel's driver The near-native (the no hypervisor's mediation) The performance's critical's workloads (the databases, the AI's accelerators)

The live migration's configuration

The following commands demonstrate the live migration's configuration and its inspection: 

# The migration's prerequisites:
# 1. The destination's host (the libvirt's daemon, the
#    compatible's CPU, the shared's storage or the
#    block's migration)
# 2. The migration's channel (the libvirt's TCP's
#    connection or the SSH)
# 3. The compatible's VM's configuration (the
#    devices' models, the network's interfaces)

# The migration's command (the SSH's transport)
virsh migrate --live --verbose --persistent \
    --undefinesource demo-vm \
    qemu+ssh://dest-host/system

# The migration's parameters:
#   --live: the live's migration
#   --persistent: the VM's definition's transfer
#   --undefinesource: the VM's definition's removal
#                      from the source
#   --verbose: the progress's display

# The migration's tuning
virsh migrate-setspeed demo-vm --bandwidth 1000  # the MB/s
virsh migrate-setspeed demo-vm --bandwidth 0     # the unlimited

# The migration's post-copy's mode
virsh migrate --live --postcopy --verbose demo-vm \
    qemu+ssh://dest-host/system

# The migration's inspection
virsh domjobinfo demo-vm 2>/dev/null

# The migration's output (the abbreviated):
# Job type:         Unbounded
# Operation:        Outgoing migration
# Time elapsed:     12.345 ms
# Data processed:   1.234 GiB
# Data remaining:   123.456 MiB
# Data total:       1.345 GiB
# Memory processed: 1.234 GiB
# Memory remaining: 123.456 MiB
# Memory total:     1.345 GiB
# Dirty rate:       12.345 MiB/s
# Iteration:        3
# Expected downtime: 45 ms

# The migration's cancellation
virsh domjobabort demo-vm 2>/dev/null

The migration's inspection's findings: the domjobinfo's output shows the migration's progress (the data processed, the dirty's rate, the iteration's count, the expected's downtime), and the observation is the reason the migration's tuning (the bandwidth's limit, the auto-convergence, the post-copy's mode) is the administrators' and the operators' ongoing's activity. The migration's downtime's typical's values: the 50–500 ms for the moderate's VMs (the 1–4 GB of the memory), and the higher for the large's VMs (the 10–100 GB of the memory) without the optimizations.

Comparing the hypervisors' designs

The following comparison summarizes the hypervisors' designs' characteristics:

The aspect The KVM The Xen The VMware ESXi
The type The type 0 (the hybrid's) The type 1 (the bare-metal's) The type 1 (the bare-metal's)
The architecture The hybrid (the kernel's module + the user-space's VMM) The microkernel (the dom0's drivers) The monolithic with the modules
The TCB's size The large (the Linux's kernel) The small (the hypervisor's core) The large (the VMkernel's + the drivers)
The performance The high (the hardware's assistance, the vhost) The moderate (the IPC's overhead) The high (the optimized's paths)
The management The libvirt, the OpenStack, the cloud's tools The XenAPI, the XenCenter, the Xen Orchestra The vSphere's API, the vCenter
The ecosystem The broad (the Linux's, the open-source's) The moderate (the Xen's, the Qubes's) The broad (the enterprise's, the third-party's)
The cost The open-source (the free) The open-source (the free) The commercial (the licenses)
The typical use The cloud's providers, the enterprise's Linux's servers The security's environments (the Qubes OS) The enterprise's virtualization

The comparison shows the hypervisors' designs' trade-offs: the KVM's hybrid's design's performance's and the ecosystem's advantages (the Linux's integration), the Xen's microkernel's design's security's advantages (the small's TCB), and the ESXi's monolithic's design's features's and the enterprise's ecosystem's advantages. The choice depends on the requirements (the performance's vs. the security's, the cost's vs. the features'), and the modern's data centers use the KVM (the cloud's providers, the OpenStack), the ESXi (the enterprise's vSphere's), and the Hyper-V (the Windows's environments) as the primary's hypervisors.

06 Interactive activity

Activity · explore the hypervisors on a live system

The goal of this activity is to use the Linux command line to inspect the hypervisor's components, to create and manage a VM, to observe the I/O's paths, and to examine the management's interfaces. Work through the steps in your course VM.

  1. Inventory the hypervisor's components. Use the lsmod, the ls, and the cat commands to inspect the KVM-based hypervisor's components. 
    # The hypervisor's modules
lsmod | grep -E 'kvm|vfio|vhost'
ls /sys/module/kvm/ 2>/dev/null
cat /sys/module/kvm_intel/parameters/nested 2>/dev/null
cat /sys/module/kvm_amd/parameters/nested 2>/dev/null

# The /dev/kvm's device
ls -la /dev/kvm
cat /proc/misc | grep kvm

# The virtualization's detection
systemd-detect-virt
lscpu | grep -E 'Virtualization|Hypervisor'
grep -E 'vmx|svm' /proc/cpuinfo | head -2

# The IOMMU's support
dmesg | grep -iE 'iommu|vt-d|dmar' | head -5
ls /sys/kernel/iommu_groups/ 2>/dev/null | head -5

# The virtualization's capabilities
virsh capabilities 2>/dev/null | head -30
    Which hypervisor's modules are loaded? Is the nested virtualization enabled? What are the CPU's virtualization's flags? Is the IOMMU enabled? Write down what you find.
  2. Install and configure the virtualization's tools. Install the libvirt's and the QEMU's packages and verify the daemon. 
    # Install the tools
sudo apt install -y qemu-kvm libvirt-daemon-system \
    libvirt-clients bridge-utils virtinst virt-manager 2>/dev/null || \
    sudo dnf install -y qemu-kvm libvirt virt-install virt-manager 2>/dev/null

# Start the libvirt's daemon
sudo systemctl enable --now libvirtd 2>/dev/null
systemctl status libvirtd 2>/dev/null | head -5

# Add the user to the libvirt's group
sudo usermod -aG libvirt,kvm $(whoami)

# Verify the installation
virsh version 2>/dev/null
virsh list --all 2>/dev/null
virsh net-list --all 2>/dev/null
virsh pool-list --all 2>/dev/null

# The libvirt's configuration
ls /etc/libvirt/
cat /etc/libvirt/libvirtd.conf 2>/dev/null | grep -vE '^#|^$' | head -10
cat /etc/libvirt/qemu.conf 2>/dev/null | grep -vE '^#|^$' | head -10
    Did the installation succeed? Is the libvirtd's daemon running? What are the default's networks and the storage's pools? What are the libvirt's configuration's settings? Write down what you find.
  3. Create and manage a VM. Create a VM with the virt-install's tool (or the virsh define's with an XML's file) and inspect its configuration. 
    # The VM's creation (the demonstration's command)
sudo virt-install \
    --name demo-vm \
    --memory 1024 \
    --vcpus 1 \
    --cpu host-passthrough \
    --disk size=5,format=qcow2,bus=virtio \
    --network network=default,model=virtio \
    --graphics none \
    --console pty,target_type=serial \
    --import \
    --noautoconsole 2>/dev/null

# The VM's inspection
virsh list --all
virsh dominfo demo-vm
virsh dumpxml demo-vm | head -50
virsh vcpuinfo demo-vm 2>/dev/null
virsh domblklist demo-vm
virsh domiflist demo-vm

# The VM's devices' inspection
virsh domblkinfo demo-vm vda 2>/dev/null
virsh domstats demo-vm 2>/dev/null | head -30

# The VM's console
virsh console demo-vm

# The VM's lifecycle's operations
virsh start demo-vm 2>/dev/null
virsh suspend demo-vm 2>/dev/null
virsh resume demo-vm 2>/dev/null
virsh shutdown demo-vm 2>/dev/null
virsh destroy demo-vm 2>/dev/null

# The VM's removal
virsh undefine demo-vm --remove-all-storage 2>/dev/null
    What is the VM's configuration (the vCPUs, the memory, the disk, the network)? How do the virtio's devices appear in the configuration? What are the VM's lifecycle's operations' effects? Write down what you find.
  4. Examine the QEMU's process and its resources. Use the ps, the lsof, and the /proc's interfaces to inspect the QEMU's process (if a VM runs). 
    # The QEMU's process
ps -ef | grep qemu | head -5
ps -eLf | grep qemu | wc -l

# The QEMU's threads
ps -eLf | grep qemu | head -10
ps -eLf | grep -E 'CPU [0-9]+/KVM' | head -5

# The QEMU's file descriptors
sudo ls -la /proc/$(pgrep -f qemu | head -1)/fd 2>/dev/null | head -20

# The QEMU's memory's maps
sudo cat /proc/$(pgrep -f qemu | head -1)/maps 2>/dev/null | head -20

# The vhost's kernel's threads
ps -ef | grep '\[vhost' | head -5

# The VM's cgroup's placement
systemd-cgls /machine.slice/ 2>/dev/null | head -20
ls /sys/fs/cgroup/machine.slice/ 2>/dev/null
    How many threads does the QEMU's process have? What are the vCPU's threads (the CPU X/KVM's)? What are the QEMU's file descriptors (the disks, the network, the KVM's device)? What are the vhost's kernel's threads? How does the VM appear in the cgroups? Write down what you find.
  5. Examine the I/O's paths. Inspect the virtio's and the vhost's configurations and the interrupt's distribution. 
    # The VM's devices' configuration
virsh dumpxml demo-vm 2>/dev/null | grep -A5 -E 'driver name|model type'

# The interrupt's distribution
grep -E 'vfio|virtio' /proc/interrupts | head -10

# The vhost's network's configuration
sudo cat /sys/class/net/*/device/uevent 2>/dev/null | head -10

# The TAP's devices
ip link show type tap 2>/dev/null
ip link show type tun 2>/dev/null

# The bridge's configuration
ip link show type bridge 2>/dev/null
bridge link show 2>/dev/null

# The nftables' rules (if the libvirt's network is used)
sudo nft list ruleset 2>/dev/null | head -20
    What are the VM's devices' models (the virtio's)? How do the interrupts appear (the virtio's or the vfio's)? What are the bridge's and the TAP's configurations? How does the virtual network work? Write down what you find.
  6. Examine the management's interfaces. Use the libvirt's tools to inspect the management's interfaces. 
    # The libvirt's networks
virsh net-list --all 2>/dev/null
virsh net-dumpxml default 2>/dev/null

# The libvirt's storage's pools
virsh pool-list --all 2>/dev/null
virsh pool-dumpxml default 2>/dev/null

# The libvirt's secrets (the encrypted's credentials)
virsh secret-list 2>/dev/null

# The libvirt's snapshots
virsh snapshot-list demo-vm 2>/dev/null
virsh snapshot-create-as demo-vm snap1 --description "The test's snapshot" 2>/dev/null
virsh snapshot-list demo-vm 2>/dev/null
virsh snapshot-info demo-vm snap1 2>/dev/null

# The libvirt's events
virsh event --list 2>/dev/null | head -10

# The libvirt's daemon's configuration
ls /etc/libvirt/qemu/ 2>/dev/null
ls /var/lib/libvirt/ 2>/dev/null
ls /var/lib/libvirt/images/ 2>/dev/null
    What are the libvirt's default's networks and the storage's pools? What does the snapshot's creation do? What are the libvirt's directories? How does the libvirt's management's interface work?
  7. Compare the hypervisors' designs. If your VM has the access to the other hypervisors (the Docker's, the Podman's, the VirtualBox's, the Hyper-V's), compare their designs: 
    # The KVM's components
lsmod | grep kvm
ls /dev/kvm

# The VirtualBox's components (if installed)
lsmod | grep vbox
ls /dev/vbox*
VBoxManage --version 2>/dev/null

# The Xen's components (if installed)
ls /dev/xen 2>/dev/null
xl info 2>/dev/null | head -10

# The hypervisor's comparison
echo "=== The available hypervisors ==="
for hv in kvm xen vbox hyperv; do
    if lsmod | grep -q "$hv" 2>/dev/null; then
        echo "$hv: the module is loaded"
    fi
done
    Which hypervisors are available on the system? How do their components differ (the kernel's modules, the device's model, the management's tools)? What does the comparison tell you about the hypervisors' designs?
  8. Read the man pages. Read the man pages that document the hypervisors' interfaces. 
    man 1 virsh
man 8 virt-install
man 1 qemu-system-x86_64
man 8 qemu-img
man 8 libvirtd
man 5 libvirt
man 5 qemu
man 5 qcow2
man 1 lscpu
man 1 systemd-detect-virt
man 1 ps
man 1 lsof
man 1 taskset
man 1 numactl
    For each, write a sentence describing its main contribution to your understanding of the hypervisors. The man 1 virsh page documents the libvirt's management's tool; the man 8 qemu-img page documents the disk's images' tool; the man 1 taskset page documents the CPU's affinity's tool (the vCPUs' pinning).

Reflect: the hypervisors are the virtualization's central's component, and their designs (the types: the bare-metal's, the hosted's, the hybrid's; the architectures: the monolithic's, the microkernel's, the hybrid's; the components: the VMM, the vCPU's scheduler, the memory's manager, the I/O's manager) determine the VMs' performance, their isolation, their compatibility, and their manageability. Which of the hypervisors' designs is the most appropriate for the modern's data centers, and why? What would you change about the hypervisors' designs if you could? How would you explain the difference between the type 1's and the type 2's hypervisors to someone who has never thought about it? What is the right balance between the hypervisor's functionality and the minimal's trusted's computing's base?

07 Knowledge check

Check your understanding

Question 1

Define the hypervisor and describe its roles (the virtual hardware's emulation, the resources' multiplexing, the lifecycle's management, the isolation, the management's interfaces). Distinguish the hypervisor from the VMM, and explain why the distinction matters.

Show answer
The hypervisor (the virtual machine monitor, the VMM) is the software layer that runs between the physical's hardware and the VMs. Its roles: the virtual hardware's emulation (the hypervisor presents the guest's operating system with the virtual hardware and emulates the hardware's behaviors: the CPU's, the memory's, the devices', the platform's), the resources' multiplexing (the hypervisor allocates the physical's resources among the VMs: the vCPUs' scheduling, the memory's management, the storage's I/O, the network's I/O), the lifecycle's management (the hypervisor creates the VMs, starts and stops them, pauses and resumes them, saves and restores their state, migrates them, and destroys them), the isolation (the hypervisor enforces the VMs' isolation: the memory's, the CPU's, the devices', the resources'), and the management's interfaces (the hypervisor exposes the APIs, the CLIs, the web's UIs that the administrators and the orchestration's platforms use). The distinction between the hypervisor and the VMM: the hypervisor is the complete's system (the VMM plus the vCPU's scheduler, the memory's manager, the I/O's manager, the management's interfaces, the security's mechanisms), and the VMM is the component that emulates the virtual hardware (the CPU's, the memory's, the devices' emulation). The distinction matters because: the hypervisor's design's decisions (the scheduler's policies, the memory's over-commitment, the I/O's paths) are the separate's concerns from the VMM's emulation's decisions (the instructions' emulation, the devices' models), and the two components can be the separate's (the KVM's kernel's module is the VMM's CPU's and memory's component, and the QEMU's user-space's process is the VMM's devices' component and the hypervisor's management's component). The observation: the classical's literature (the Goldberg's survey, the Popek and Goldberg's paper) uses the VMM's term, and the modern's usage prefers the hypervisor's term, and the two refer to the same software in the modern's context.

Question 2

Compare the hypervisor's types (the type 1 bare-metal, the type 2 hosted, the type 0 hybrid) and its architectures (the monolithic, the microkernel, the hybrid). What are each design's advantages and disadvantages, and when is each appropriate?

Show answer
The hypervisor's types (the position in the software's stack): the type 1 (the bare-metal's) runs directly on the physical's hardware, and the guest's operating systems run on the hypervisor; the advantages: the performance (the direct's access), the security (the small's TCB), the reliability; the disadvantages: the hardware's compatibility (the limited's drivers), the management (the specialized's skills); the examples: the VMware's ESXi, the Xen, the Hyper-V's hypervisor, the z/VM. The type 2 (the hosted's) runs as an application on the host's operating system, and the guest's operating systems run on the hypervisor; the advantages: the compatibility (the broad's drivers), the convenience (the application's installation), the functionality (the host's OS's services); the disadvantages: the performance (the host's OS's overhead), the security (the host's OS's attack surface); the examples: the VMware's Workstation and Fusion, the VirtualBox, the QEMU's user's mode. The type 0 (the hybrid's) integrates the hypervisor and the kernel; the advantages: the performance (the kernel's integration), the integration (the kernel's services), the management (the standard's tools); the disadvantages: the kernel's attack surface, the complexity (the two's components); the examples: the KVM, the bhyve, the Hyper-V's aspects. The hypervisor's architectures (the internal's design): the monolithic (the single, large's software: the VMM, the scheduler, the memory's manager, the I/O's manager, the drivers, the management's interfaces); the advantages: the performance (the direct's communication), the simplicity (the single's codebase); the disadvantages: the large's TCB, the reliability (the driver's bug crashes the whole); the examples: the ESXi's, the KVM's aspects. The microkernel (the minimal's core plus the separate's components: the domains or the service's VMs); the advantages: the small's TCB (the easier's verification), the reliability (the driver's isolation), the security (the components' isolation); the disadvantages: the performance (the IPC's overhead), the complexity (the components' communication); the examples: the Xen, the seL4. The hybrid (the combination: the performance's critical's components in the core, the other's components the separate's or the modules); the advantages: the performance and the modularity; the disadvantages: the complexity (the boundaries' design); the examples: the KVM, the Hyper-V, the ESXi's modular's aspects. The designs' appropriateness: the type 1 and the type 0 for the production's environments (the performance's and the security's requirements); the type 2 for the development and the testing (the convenience's and the compatibility's requirements); the monolithic for the performance's priorities; the microkernel for the security's priorities (the small's TCB); the hybrid for the balance (the modern's standard: the KVM).

Question 3

Describe the major hypervisors (the KVM, the Xen, the VMware ESXi, the Microsoft Hyper-V, the QEMU, the Firecracker, the bhyve). What is each one's design, its type, its architecture, its advantages, and its typical use?

Show answer
The major hypervisors: the KVM (the Linux's kernel's virtualization's module; the type 0's hybrid's; the kernel's modules (the kvm.ko, the kvm_intel.ko or the kvm_amd.ko) provide the virtualization's interfaces (the /dev/kvm's device, the KVM_RUN's ioctl, the memory's slots), and the user-space's program (the QEMU, the Firecracker, the crosvm) provides the device's model and the lifecycle's management; the advantages: the performance (the kernel's integration, the hardware's assistance), the integration (the Linux's services), the flexibility (the user-space's VMMs); the disadvantages: the kernel's attack surface, the management's complexity; the typical use: the cloud's providers (the OpenStack, the GCP), the enterprise's Linux's servers). the Xen (the Cambridge's microkernel's hypervisor; the type 1's bare-metal's; the hypervisor's core (the VMM, the scheduler, the memory's manager), the dom0's privileged's domain (the drivers, the management), the domU's guest's domains, the paravirtualization's or the HVM's modes; the advantages: the security (the small's TCB), the isolation (the driver's domain's separation), the scalability; the disadvantages: the performance (the IPC's overhead), the management's complexity; the typical use: the security's environments (the Qubes OS), the AWS's early's infrastructure). the VMware ESXi (the VMware's type 1's hypervisor; the VMkernel (the proprietary's kernel), the per-VM's VMM, the vCenter (the management's platform); the advantages: the performance (the mature's design), the features (the vMotion, the DRS, the HA, the FT), the ecosystem (the vSphere); the disadvantages: the cost, the vendor's lock-in, the closed's design; the typical use: the enterprise's virtualization, the private's clouds). the Microsoft Hyper-V (the Microsoft's hypervisor; the type 1's with the type 0's aspects; the hypervisor's core (the minimal's), the parent's partition (the Windows's with the virtualization's stack), the child's partitions; the advantages: the integration (the Windows's ecosystem), the features (the live's migration, the replicas, the shielded's VMs), the licensing (the included with the Windows Server); the disadvantages: the Windows's dependency, the performance (the parent's partition's overhead); the typical use: the Windows's environments, the Microsoft's clouds (the Azure)). the QEMU (the user-space's emulator and the VMM's component; the type 2's user-space's VMM or the KVM's user-space's component; the device's model (the many's emulated's devices), the TCG's (the dynamic's binary translation), the KVM's mode; the advantages: the flexibility (the many's devices and architectures), the openness, the integration (the libvirt, the KVM); the disadvantages: the performance (the emulation's overhead), the security (the device's model's vulnerabilities: the VENOM's); the typical use: the development, the testing, the KVM's device's model). the Firecracker (the AWS's microVM's hypervisor; the type 0's KVM's based; the minimal's VMM (the Rust's implementation: the vCPUs, the memory, the minimal's virtio's devices), the KVM's kernel's module, the jailer (the sandboxing's component); the advantages: the fast's startup (the 125 ms), the low's memory's overhead (the ~5 MB), the strong's isolation (the VM's boundary), the minimal's attack surface; the disadvantages: the limited's devices, the specific's use's case; the typical use: the serverless's and the container's workloads (the AWS's Lambda, the Fargate)). the bhyve (the FreeBSD's hypervisor; the type 2's with the type 1's aspects; the kernel's module (the VMM, the vCPUs, the memory), the bhyve's command, the bhyvectl's tool; the advantages: the FreeBSD's integration (the ZFS, the jails, the pf), the performance, the licensing (the BSD's); the disadvantages: the FreeBSD's dependency, the ecosystem (the fewer's tools); the typical use: the FreeBSD's environments).

Question 4

Describe the hypervisor's components (the VMM, the vCPU's scheduler, the memory's manager, the I/O's manager, the device's model, the timers and the interrupt's controller) and the I/O's architecture (the QEMU's device's model, the virtio, the vhost, the vhost-user, the passthrough). How do the components affect the virtualization's performance?

Show answer
The hypervisor's components: the VMM (the virtual hardware's emulation: the CPU's emulation (the traps and the emulation, the hardware's assistance, the guest's context's management), the memory's virtualization (the address's translation, the EPT's or the NPT's management, the shadow's page's tables), the devices' emulation (the emulated's devices, the virtio's backends, the passthrough's devices)); the vCPU's scheduler (the physical's CPUs' time's allocation among the vCPUs: the policies (the Linux's CFS for the KVM, the credit's scheduler for the Xen, the ESXi's proportional's shares), the co-scheduling (the SMP's guests), the vCPUs' migration, the over-commitment, the latency's considerations); the memory's manager (the GPA's to the HPA's mapping, the over-commitment (the ballooning, the page's sharing, the compression, the swapping), the memory's hot-plug, the NUMA's awareness); the I/O's manager (the device's model, the virtio's backends, the passthrough's devices, the virtual switches, the storage's stack); the device's model (the emulated's devices: the QEMU's (the IDE's, the SCSI's, the NVMe's, the network's, the USB's, the VGA's, the others), the virtio's devices (the virtio-net, the virtio-blk, the virtio-scsi, the virtio-gpu, the virtio-fs, the virtio-vsock), the platform's devices (the APIC, the timers, the HPET)); the timers and the interrupt's controller (the timers (the PIT, the HPET, the local APIC's timer, the TSC), the interrupt's controller (the PIC, the IOAPIC, the MSI's and the MSI-X's emulation, the posted's interrupts), the clock's sources (the kvm-clock's paravirtualized's clock)). The I/O's architecture: the QEMU's user-space's device's model (the classical's design: the QEMU's process emulates the devices, and the guest's I/O's requests are the trapped and the handled in the user-space; the advantages: the flexibility, the isolation; the disadvantages: the performance (the 10–30% for the I/O's intensive's workloads)); the virtio (the paravirtualized's interfaces: the guest's drivers and the hypervisor's backends communicate through the shared's memory's rings (the virtqueues) and the notifications; the advantages: the performance (the batched's requests, the shared's memory), the standardization (the OASIS's specification); the virtio's devices: the virtio-net, the virtio-blk, the virtio-scsi, the virtio-gpu, the virtio-fs, the virtio-vsock); the vhost (the kernel's acceleration's framework: the vhost's kernel's module (the vhost-net, the vhost-scsi) handles the virtio's queues in the kernel; the advantages: the performance (the kernel's processing: the no user-space's context's switches)); the vhost-user (the user-space's acceleration's framework: the dedicated's user-space's daemon (the OVS's vhost-user, the SPDK's vhost, the virtiofsd) handles the virtio's queues in the user-space; the advantages: the flexibility, the isolation, the performance (the shared's memory: the no copies)); and the passthrough (the device's assignment: the hypervisor assigns the physical's device directly to the guest's VM (the VFIO's framework, the SR-IOV's virtual functions); the advantages: the near-native's performance, the compatibility; the disadvantages: the exclusivity, the migration's difficulty, the security's considerations (the IOMMU's isolation: the mandatory's requirement)). The components' performance's effects: the CPU's and the memory's overhead is the minimal (the 1–5% with the hardware's assistance), the I/O's overhead is the higher without the paravirtualized's drivers (the 10–30%), and the virtio, the vhost, the vhost-user, and the passthrough reduce the I/O's overhead to the near-native (the 2–8%). The observation: the hypervisor's components' design determines the virtualization's performance, and the modern's hypervisors combine the hardware's assistance with the paravirtualized's drivers and the vhost's frameworks.

Question 5

Describe the hypervisor's security (the attack surface, the VM's escapes, the side channels, the confidential computing), the live migration's mechanisms, the nested virtualization, and the microVMs. What are the trade-offs and the trends?

Show answer
The hypervisor's security: the attack surface (the hypervisor's code: the VMM, the device's model, the drivers; the management's interfaces: the APIs, the daemons; the configuration: the shared's resources, the passthrough's devices, the live's migration's channels; the smaller the code, the smaller the attack surface, and the minimal's hypervisors (the Xen's, the Firecracker's, the seL4's) are the high-security's preference); the VM's escapes (the attacker's exploit of the hypervisor's or the guest's vulnerability to break the isolation: the classical's examples: the VENOM (the CVE-2015-3456: the QEMU's floppy's controller's buffer's overflow), the Xen's XSA's advisories, the CVE-2021-3653 and the CVE-2021-3656 (the AMD's SEV's); the mitigations: the hypervisor's updates, the minimal's configurations, the hardware's assistance (the IOMMU, the EPT, the NPT), the monitoring, the confidential computing); the side channels (the information's leaks through the shared's resources: the CPU's caches, the memory's deduplication (the KSM), the speculative's execution (the Spectre's and the Meltdown's), the power's, the electromagnetic's; the mitigations: the caches' partitioning, the KSM's disabling, the microcode's updates, the speculative's execution's mitigations, the constant-time's programming); the confidential computing (the AMD's SEV, the SEV-ES, the SEV-SNP, the Intel's SGX, the TDX, the ARM's CCA; the advantages: the protection from the hypervisor's compromise, the compliance's requirements; the disadvantages: the performance (the 5–15%), the limited's features, the ecosystem). The live migration: the pre-copy's phase (the memory's pages are the copied while the VM runs, and the subsequent's passes copy the dirty's pages), the stop-and-copy's phase (the VM is the paused, the remaining's pages and the state are the copied, and the VM starts on the destination: the downtime's 50–500 ms for the typical's VMs), the post-copy's phase (the alternative: the VM starts on the destination, and the remaining's pages are the fetched on the demand), the optimizations (the compression, the deduplication, the multi-threading, the RDMA, the auto-convergence), and the constraints (the compatible's CPUs, the shared's storage, the network's configuration, the compatibility). The nested virtualization: the hypervisor's capability to run the VMs inside the VMs (the L0's hypervisor, the L1's VM that is the hypervisor, the L2's VMs); the mechanisms: the hardware's assistance's extensions, the L0's emulation, the multiple's translation's levels (the 10–30% overhead for the L2's VMs); the uses: the development, the cloud's services, the security's research. The microVMs: the lightweight's VMs designed for the serverless's and the container's workloads (the AWS's Firecracker, the Google's gVisor, the Kata Containers, the QEMU's microvm's machine's type); the advantages: the strong's isolation (the VM's boundary), the fast's startup (the tens of the milliseconds), the minimal's attack surface; the disadvantages: the limited's devices, the specific's use's case. The trends: the hardware's assistance's completeness, the hypervisor's specialization (the microVMs, the unikernels, the real-time's, the embedded's), the cloud's native's hypervisors (the AWS's Nitro, the Google's, the Microsoft's, the Alibaba's Dragonfly's), the container's integration (the Kata Containers, the KubeVirt), and the confidential computing's adoption.

08 Unit discussion

Discussion prompt

The hypervisors are the virtualization's central's component, and their history is the history of the virtualization's evolution. The classical's hypervisors (the IBM's mainframes' CP/CMS and VM/370 in the 1960s and the 1970s) were the type 1's bare-metal's designs, and their trap-and-emulate's technique and the small's TCB were the virtualization's foundation. The modern's hypervisors (the VMware's ESX in 2001, the Xen's paravirtualization in 2003, the KVM's kernel's integration in 2007, the Hyper-V in 2008) brought the virtualization to the x86's servers, and their designs (the monolithic's, the microkernel's, the hybrid's) reflected the different's priorities (the performance's, the security's, the integration's). The modern's hypervisors (the KVM's, the Xen's, the ESXi's, the Hyper-V's, the Firecracker's) are the data center's and the cloud's foundation, and the observation is the reason the hypervisors' designs are the modern's computing's central's concerns.

The trend is toward the minimalism and the specialization. The classical's hypervisors were the general-purpose's (the many's devices, the many's features); the modern's hypervisors include the minimal's designs (the Firecracker's: the ~50,000's lines of the Rust's code, the ~5 MB's memory's overhead, the 125 ms's boot's time; the gVisor's: the user-space's kernel; the unikernels: the single-address-space's kernels), and the observation is the reason the minimalism is the modern's hypervisors' design's trend: the smaller the code, the smaller the attack surface, the faster the startup, and the lower the overhead.

The trend also raises a question about the boundaries between the hypervisors and the containers. The VMs' advantages (the strong's isolation, the separate's kernels) and the containers' advantages (the lightweight's resource's usage, the fast's startup, the density) are the different's trade-offs, and the modern's systems combine the two (the Kata Containers, the Firecracker's microVMs, the gVisor's user-space's kernel, the KubeVirt's VMs in the Kubernetes's clusters). The observation is the reason the VMs' and the containers' boundaries are the blurring, and the observation is the reason the modern's platforms support the both.

Which of the hypervisors' designs (the type 1's, the type 2's, the type 0's; the monolithic's, the microkernel's, the hybrid's) is the most appropriate for the modern's data centers, and why? Should the hypervisors' design prioritize the performance (the hardware's assistance, the passthrough's devices) or the security (the small's TCB, the confidential computing)? What is the right balance between the hypervisors' functionality and the minimalism? How should the hypervisors' design evolve for the confidential computing, the serverless's platforms, and the edge's computing? Post your response to the Unit 13 discussion forum and reply to at least two classmates. Supports CLO1, CLO7, and CLO8.

09 Practice exercise

This ungraded exercise consolidates the lesson and prepares you for Lesson 13.3. Aim to spend 60–90 minutes on it.

  1. Compare the hypervisors. Write a short essay (400–500 words) comparing the hypervisors (the KVM, the Xen, the VMware ESXi, the Microsoft Hyper-V, the QEMU, the Firecracker). Your essay should address:
    • Each hypervisor's type (the type 1's, the type 2's, the type 0's).
    • Each hypervisor's architecture (the monolithic's, the microkernel's, the hybrid's).
    • Each hypervisor's TCB's size and its security's implications.
    • Each hypervisor's performance's characteristics.
    • Each hypervisor's management's interfaces and its ecosystem.
    • Each hypervisor's appropriate use's cases.
    Conclude with a recommendation for a general-purpose's data center, for a high-security's environment, and for a serverless's platform.
  2. Trace a VM's lifecycle. A VM's lifecycle includes the creation, the start, the execution, the snapshot, the live migration, and the stop. For each stage, describe:
    • The hypervisor's components involved.
    • The resources allocated or released.
    • The state changes.
    • The potential's failure's modes.
    Which stage has the highest overhead? Which has the most failure's modes? Explain.
  3. Design a hypervisor's configuration. A company is building a virtualization's infrastructure for its production's workloads (the web servers, the databases, the AI's inference's servers). The hosts have the 64 cores, the 512 GB of the RAM, the NVMe's storage, and the 100 GbE's network. Design the hypervisor's configuration:
    • The hypervisor's selection (the KVM, the Xen, the ESXi, the Hyper-V) and its rationale.
    • The type (the type 1's, the type 2's, the type 0's) and the architecture's considerations.
    • The vCPUs' and the memory's allocation (the over-commitment's ratios, the NUMA's awareness).
    • The I/O's configuration (the virtio's devices, the vhost's framework, the passthrough's devices, the SR-IOV).
    • The storage's configuration (the shared's storage, the images' formats).
    • The network's configuration (the bridges, the Open vSwitch, the SR-IOV's VFs).
    • The security's configuration (the minimal's hypervisor's components, the updates, the monitoring, the confidential computing).
    • The management's tools (the libvirt, the OpenStack, the oVirt).
    Justify your design through the lesson's principles.
  4. Explore the hypervisor's components. On your VM, explore the KVM-based hypervisor's components: 
    # The hypervisor's modules
lsmod | grep -E 'kvm|vfio|vhost'
ls /sys/module/kvm/
cat /sys/module/kvm_intel/parameters/nested 2>/dev/null

# The libvirt's capabilities
virsh capabilities 2>/dev/null | head -40

# The QEMU's process (if a VM runs)
ps -ef | grep qemu
sudo ls -la /proc/$(pgrep -f qemu | head -1)/fd 2>/dev/null | head -20

# The vhost's kernel's threads
ps -ef | grep '\[vhost'

# The VM's cgroup's placement
systemd-cgls /machine.slice/ 2>/dev/null | head -20

# The interrupt's distribution
grep -E 'vfio|virtio' /proc/interrupts | head -10
    What are the hypervisor's modules and parameters? What are the QEMU's processes and threads? What are the vhost's threads? How does the VM appear in the cgroups? Write down what you find.
  5. Compare the I/O's paths. If your VM has the access to the different I/O's paths, measure their performance: 
    # The QEMU's user-space's virtio-blk
# (the configuration: the QEMU's default)

# The vhost-blk
# (the configuration: the QEMU's vhost=on)

# The vhost-user-blk
# (the configuration: the SPDK's or the
# virtiofsd's backend)

# The VFIO's passthrough
# (the configuration: the device's assignment)

# The measurement (the fio's benchmark)
fio --name=randread --filename=/dev/vdX --size=1G --rw=randread \
    --bs=4k --iodepth=64 --direct=1 --runtime=10 2>/dev/null | grep bw
    Which path has the highest performance? Which has the lowest? How do the paths' overheads compare with the lesson's values? Explain the differences.
  6. Design a live migration's plan. A company needs to migrate the 100 VMs (the average: the 4 vCPUs, the 8 GB of the RAM, the 100 GB of the disk) from the old's hosts to the new's hosts with the minimal's downtime. Design the migration's plan:
    • The migration's technique (the pre-copy's, the post-copy's, the hybrid's).
    • The migration's bandwidth's allocation.
    • The migration's scheduling (the batches, the priorities, the windows).
    • The storage's strategy (the shared's storage or the block's migration).
    • The compatibility's checks (the CPUs' features, the devices' models).
    • The rollback's plan (the failures' handling).
    • The monitoring and the verification.
    Justify your plan through the lesson's principles.
  7. Analyze a hypervisor's security's incident. Choose a known hypervisor's security's incident (the VENOM's vulnerability in the QEMU's floppy's controller, the Xen's XSA's advisories, the CVE-2021-3653's in the AMD's SEV, the VirtualBox's escapes, the CVE-2020-14364's in the QEMU's USB's controller) and analyze it:
    • The vulnerability's description.
    • The escape's mechanism (which component: the VMM's, the device's model's, the management's).
    • The affected's versions and the fixes.
    • The lessons for the hypervisors' security's design.
    • The mitigations (the minimal's hypervisors, the hardware's assistance, the monitoring).
    Write a short report (500–750 words).
  8. Compare the microVMs and the containers. Write a short paragraph (200–250 words) comparing the microVMs (the Firecracker's, the gVisor's, the Kata Containers') and the containers (the Docker's, the Podman's). Your paragraph should address:
    • The isolation's mechanisms (the VM's boundary vs. the namespaces and the cgroups).
    • The startup's time (the milliseconds vs. the hundreds of the milliseconds).
    • The memory's overhead (the MBs vs. the MBs).
    • The density (the hundreds vs. the thousands per host).
    • The security's characteristics (the separate's kernel vs. the shared's kernel).
    • The use's cases (the serverless's vs. the general-purpose's).
    • The combinations (the Kata Containers, the AWS's Fargate).
  9. Reflect. In your own words, explain the hypervisor's role and its types (the type 1's, the type 2's, the type 0's) and architectures (the monolithic's, the microkernel's, the hybrid's), and describe the major hypervisors and their designs. Then describe one situation in your own experience — a VM you have used, a cloud's service you have relied on, or a virtualization's environment you have administered — where the hypervisor's design affected the computing. What was the situation, and how would a different hypervisor or a different configuration have changed the outcome?

10 Summary

  • The hypervisor (the virtual machine monitor, the VMM) is the software layer that runs between the physical's hardware and the VMs. Its roles: the virtual hardware's emulation, the resources' multiplexing, the lifecycle's management, the isolation, and the management's interfaces. The hypervisor is the complete's system; the VMM is the emulation's component.
  • The hypervisor's types: the type 1 (the bare-metal's) (the hypervisor runs directly on the hardware: the ESXi, the Xen, the Hyper-V's hypervisor), the type 2 (the hosted's) (the hypervisor runs on the host's operating system: the Workstation, the VirtualBox, the QEMU's user's mode), and the type 0 (the hybrid's) (the hypervisor and the kernel are the integrated: the KVM, the bhyve).
  • The hypervisor's architectures: the monolithic (the single, large's software: the fast's paths, the large's TCB: the ESXi, the KVM's aspects), the microkernel (the minimal's core plus the separate's components: the small's TCB, the IPC's overhead: the Xen, the seL4), and the hybrid (the combination: the performance's critical's components in the core, the other's separate's: the KVM, the Hyper-V).
  • The major hypervisors: the KVM (the type 0's, the hybrid's: the kernel's modules + the user-space's VMM: the cloud's providers), the Xen (the type 1's, the microkernel's: the security's environments), the VMware ESXi (the type 1's, the monolithic's with the modules: the enterprise's virtualization), the Microsoft Hyper-V (the type 1's with the type 0's aspects: the Windows's environments), the QEMU (the user-space's emulator and the VMM's component: the development, the KVM's device's model), the Firecracker (the microVM's: the serverless's and the container's workloads), and the bhyve (the FreeBSD's).
  • The hypervisor's components: the VMM (the CPU's, the memory's, the devices' emulation), the vCPU's scheduler, the memory's manager (the EPT's and the NPT's mapping, the ballooning, the KSM), the I/O's manager, the device's model, the timers and the interrupt's controller.
  • The I/O's architecture: the QEMU's user-space's device's model, the virtio (the paravirtualized's interfaces: the virtqueues, the OASIS's specification), the vhost (the kernel's acceleration's framework), the vhost-user (the user-space's acceleration's framework), and the passthrough (the VFIO's framework, the SR-IOV).
  • The management's interfaces: the libvirt (the unified's API: the virsh, the virt-install, the virt-manager), the vSphere's API (the VMware's), the WMI and the PowerShell (the Hyper-V's), the XenAPI and the XenServer, and the OpenStack (the cloud's platform).
  • The hypervisor's security: the attack surface, the VM's escapes (the VENOM, the Xen's XSA's, the AMD's SEV's vulnerabilities), the side channels (the caches, the KSM, the speculative's execution), and the confidential computing (the AMD's SEV, the Intel's SGX and TDX, the ARM's CCA).
  • The live migration: the pre-copy's phase, the stop-and-copy's phase (the downtime: the 50–500 ms), the post-copy's phase, the optimizations (the compression, the deduplication, the RDMA, the auto-convergence), and the constraints (the compatible's CPUs, the shared's storage, the network's configuration).
  • The nested virtualization (the VMs inside the VMs: the L0, the L1, the L2: the 10–30% overhead) and the microVMs (the Firecracker's, the gVisor's, the Kata Containers': the strong's isolation with the fast's startup). The trends: the hardware's assistance's completeness, the hypervisor's specialization, the cloud's native's hypervisors, the container's integration, and the confidential computing's adoption.

Looking ahead

This lesson presented the hypervisors: the hypervisor's role and the terminology, the types (the type 1's, the type 2's, the type 0's), the architectures (the monolithic's, the microkernel's, the hybrid's), the major hypervisors (the KVM, the Xen, the ESXi, the Hyper-V, the QEMU, the Firecracker, the bhyve), the components (the VMM, the vCPU's scheduler, the memory's manager, the I/O's manager, the device's model), the I/O's architecture (the virtio, the vhost, the vhost-user, the passthrough), the management's interfaces (the libvirt, the vSphere's API, the WMI, the XenAPI, the OpenStack), the security's considerations (the escapes, the side channels, the confidential computing), the live migration, the nested virtualization, and the microVMs. It referred several times to the containers — the lightweight's isolation's mechanism that shares the host's kernel — and to the containers' and the VMs' convergence (the Kata Containers, the Firecracker's microVMs, the gVisor's user-space's kernel, the KubeVirt's VMs in the Kubernetes's clusters). Lesson 13.3 presents the containers in detail: the containers' concept and their history (the chroot, the jails, the Solaris's zones, the Linux's namespaces and cgroups), the containers' mechanisms (the namespaces, the cgroups, the union file systems, the seccomp, the capabilities), the containers' runtimes (the OCI's runtime's specification, the runc, the crun, the containerd, the CRI-O), the containers' images (the OCI's image's specification, the layers, the registries), the containers' and the VMs' comparison, and the containers' security. The lesson is the third of the unit's presentation of the virtualization's continuum, and it is the bridge to the Docker (Lesson 13.4) and the container orchestration (Lesson 13.5).

11 References

  • Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 18.
  • Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, the Virtualization chapters. Available online at ostep.org.
  • Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 7.
  • Smith, J. E., & Nair, R. The Architecture of Virtual Machines. IEEE Computer, 2005.
  • Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., & Warfield, A. Xen and the Art of Virtualization. SOSP, 2003.
  • Kivity, A., Kamay, Y., Laor, D., Lublin, U., & Liguori, A. KVM: the Linux Virtual Machine Monitor. Ottawa Linux Symposium, 2007.
  • Waldspurger, C. A. Memory Resource Management in VMware ESX Server. OSDI, 2002.
  • Habib, I. Virtualization with KVM. Packt, 2008.
  • Chirammal, H. D., Mukhedkar, P., & Vettathu, A. Mastering KVM Virtualization. Packt, 2016.
  • Agache, A., Brooker, M., Iordache, A., Liguori, A., Neugebauer, R., Piwonka, P., & Popa, D. M. Firecracker: Lightweight Virtualization for Serverless Applications. NSDI, 2020.
  • The libvirt's documentation. The libvirt's API's reference, the virsh's manual, and the domain's XML's schema.
  • Linux man-pages: man 1 virsh, man 8 virt-install, man 1 qemu-system-x86_64, man 8 qemu-img, man 8 libvirtd, man 5 libvirt, man 5 qemu, man 5 qcow2, man 1 lscpu, man 1 systemd-detect-virt, man 1 ps, man 1 lsof, man 1 taskset, man 1 numactl, man 1 perf.
👤