Containers

01 Introduction

The previous lessons presented the virtualization's approach: the virtual machines emulate the complete's hardware, and the hypervisors run the separate's kernels. The virtualization's benefits (the isolation, the consolidation, the portability) are the reasons the modern's data centers are the virtualized. But the virtualization's design's cost: the separate's kernels (each VM runs its own operating system: the GBs of the memory, the seconds of the startup, the tens of the VMs per host), and the emulation's overhead (the hypervisor's traps, the I/O's mediation). This lesson presents the containers: the lightweight's isolation that runs the processes on the shared kernel, and that achieves the isolation, the resource's limits, and the file system's packaging through the kernel's mechanisms (the namespaces, the cgroups, the union file systems, the capabilities, the seccomp, the LSM).

A container is the isolated's environment in which one or more processes run, with the isolated's view of the system's resources (the file system, the processes, the network, the IPC, the users), the limited's resource's usage (the CPU, the memory, the I/O, the processes' count), and the packaged's file system (the container's image: the application and its dependencies). The container's design's central's observation: the container's processes run on the host's kernel, and the kernel's mechanisms (the namespaces and the cgroups) provide the isolation and the limits. The observation is the reason the containers are the lightweight: the container's startup is the milliseconds (the process's start, not the kernel's boot), the container's memory's overhead is the MBs (the application's memory, not the kernel's), and the container's density is the hundreds or the thousands per host (not the tens).

The container's and the VM's comparison (the Lesson 13.1's discussion):

The containers' benefits: the lightweight (the fast's startup, the low's overhead, the high's density), the portability (the images: the application and its dependencies in the single's artifact), the consistency (the same's image runs the same's way on the developer's laptop and the production's cluster: the "it works on my machine" problem's solution), the agility (the fast's deployment, the easy's rollback, the frequent's releases), the efficiency (the resource's usage: the containers share the host's kernel, so the overhead is the minimal), and the ecosystem (the Docker's, the Kubernetes's, the cloud's providers' tools). The containers' design's trade-offs: the shared's kernel (the kernel's vulnerabilities affect the every container: the isolation is the weaker than the VMs'), the same's kernel's requirement (the Linux's containers run on the Linux's host: the different's operating systems require the VMs), and the complexity (the containers' orchestration, the networking, the storage, the security).

The containers' history: the container's concept has the deep's roots (the chroot's in the 1979's Unix, the BSD's jails in the 2000's, the Solaris's Zones in the 2004's, the Linux's OpenVZ and the Linux-VServer in the 2000s), and the modern's containers began with the LXC (the Linux Containers: the 2008's, the namespaces and the cgroups' integration), and exploded with the Docker (the 2013's, the images, the registries, the user-friendly's tooling), the OCI (the 2015's standards: the Runtime's Specification, the Image's Specification), the containerd and the CRI-O (the runtimes), and the Kubernetes (the 2014's orchestration). The observation: the containers' explosion was the Docker's user-experience's and the Kubernetes's orchestration's driver, not the kernel's mechanisms' (the namespaces and the cgroups were the years' old), and the observation is the reason the container's history is the tooling's history.

This lesson presents the containers. It defines the container and the container's comparison with the VM. It presents the container's history and the container's mechanisms (the namespaces, the cgroups, the union file systems, the capabilities, the seccomp, the LSM). It presents the OCI standards (the Runtime's Specification, the Image's Specification, the Distribution's Specification), the container's runtimes (the runc, the crun, the containerd, the CRI-O), and the container's images (the layers, the overlayfs, the registries). It presents the container's lifecycle, the container's security, and the containers' use's cases. It closes with the containers' design's trade-offs and the lesson's synthesis.

02 Learning objectives

This lesson contributes to U13.3 and U13.4, and to course learning outcomes CLO1, CLO7, and CLO8.

03 Required reading

• Operating System Concepts — Chapter 18, Virtual Machines, the section on the containers and the operating-system-level virtualization. The standard treatment of the container's concept and the container's and the VM's comparison.
• OSTEP — the Virtualization chapters, particularly the discussion of the containers and the namespaces. OSTEP's treatment of the container's mechanisms and the container's and the VM's comparison is the clearest available account, and it is the basis for the worked example in this lesson.
• Tanenbaum & Bos — Modern Operating Systems, Chapter 7, Virtualization and the Cloud. A complementary treatment that develops the container's mechanisms from first principles.
• Kerrisk, M. — The Linux Programming Interface. No Starch Press, 2010, and the man 7 namespaces, the man 7 cgroups, and the man 2 clone pages. The authoritative references for the Linux's namespaces and the cgroups.
• Rosen, R. — Resource Management: The Linux and the SUSE Approaches. The authoritative references for the cgroups.
• Merkel, D. — Docker: Lightweight Linux Containers for Consistent Development and Deployment. Linux Journal, 2014. The paper that popularized the Docker's approach.
• Bernstein, D. — Containers and Cloud: From LXC to Docker to Kubernetes. IEEE Cloud Computing, 2014. The survey of the containers' evolution.
• Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. — Borg, Omega, and Kubernetes. ACM Queue, 2016. The paper that describes the Kubernetes's design and the container's orchestration.
• The Open Container Initiative (OCI)'s specifications: the Runtime's Specification, the Image's Specification, and the Distribution's Specification. The authoritative references for the containers' standards.
• Rice, L. — Container Security. O'Reilly, 2020. The comprehensive's treatment of the container's security's mechanisms and the escapes' attacks.
• Hertz, B. — Container Security: Fundamental and the NIST SP 800-190 (the Application Container Security Guide). The security's references for the containers.
• The Linux man pages: man 7 namespaces, man 2 clone, man 2 unshare, man 1 unshare, man 1 nsenter, man 7 cgroups, man 1 systemd-cgls, man 1 systemd-cgtop, man 1 systemd-run, man 5 systemd.resource-control, man 2 seccomp, man 7 capabilities, man 8 setcap, man 5 overlay (the overlayfs's documentation), man 8 mount, man 8 podman, man 1 podman-run, man 1 podman-exec, man 1 podman-build, man 1 buildah, man 1 skopeo, man 1 docker, man 1 docker-run, man 1 runc, man 1 crun, man 1 ctr, man 1 crictl, man 1 systemd-nspawn, man 1 machinectl, man 1 lsns, man 1 dmesg, man 5 proc (for the /proc/PID/ns/'s and the /proc/PID/cgroup's interfaces).

04 Core content

4.1 What a container is

A container is the isolated's environment in which one or more processes run, with the isolated's view of the system's resources, the limited's resource's usage, and the packaged's file system. The container's design's central's observation: the container's processes run on the host's kernel, and the kernel's mechanisms provide the isolation and the limits.

The container's components:

• The isolated's views (the namespaces). The container's processes see the isolated's views of the file system (the mount's namespace), the process's table (the PID's namespace), the network (the network's namespace), the IPC (the IPC's namespace), the users (the user's namespace), the hostname (the UTS's namespace), the cgroups (the cgroup's namespace), and the time (the time's namespace).
• The limited's usage (the cgroups). The container's processes' resource's usage is the limited: the CPU's shares and quotas (the cpu's controller), the memory's limits (the memory's controller), the I/O's weights and limits (the io's controller), the processes' count (the pids's controller), and the others (the cpuset, the hugetlb, the rdma).
• The packaged's file system (the image). The container's file system is the container's image: the layered's file system (the layers are the application and its dependencies), and the container's writes go to the writable's layer (the overlayfs's upper's directory), so the image's layers are the read-only and the container's changes are the contained.
• The restricted's privileges (the capabilities, the seccomp, the LSM). The container's processes' privileges are the restricted: the capabilities' set (the default's dropped's capabilities and the added's capabilities), the seccomp's filter (the system calls' whitelist or blacklist), the LSM's profile (the SELinux's container_t, the AppArmor's docker-default), and the no-new-privileges's setting (the setuid's and the setgid's programs' prevention).
• The lifecycle's management (the runtime, the engine). The container's lifecycle is the managed by the runtime (the runc, the crun, the containerd, the CRI-O) and the engine (the Docker, the Podman, the Kubernetes's kubelet): the creation, the start, the execution, the pause, the stop, the removal.

The containers' benefits: the lightweight (the fast's startup: the milliseconds; the low's memory's overhead: the MBs; the high's density: the hundreds to the thousands per host), the portability (the images: the application and its dependencies in the single's artifact: the "it works on my machine" problem's solution), the consistency (the same's image runs the same's way in the development, the testing, the production), the agility (the fast's deployment, the easy's rollback, the frequent's releases), the efficiency (the resource's usage: the containers share the host's kernel), and the ecosystem (the Docker's, the Kubernetes's, the cloud's providers' tools). The containers' design's trade-offs: the shared's kernel (the kernel's vulnerabilities affect the every container: the isolation is the weaker than the VMs'), the same's kernel's requirement (the different's operating systems require the VMs), and the complexity (the containers' orchestration, the networking, the storage, the security).

4.2 The containers' history

The containers' history is the long's evolution, from the classical's chroot to the modern's Docker and Kubernetes:

• The chroot (the 1979's Unix). The chroot's system call changes the process's root's directory, so the process sees the specified's directory as the root. The chroot's purpose was the testing and the build's environments (the Bell Labs' Unix), and its design's weakness (the chroot is not the security's boundary: the escapes are the known) is the Lesson 12.6's discussion.
• The BSD's jails (the 2000). The FreeBSD's jails: the jail's system call (the FreeBSD 4.0's) extends the chroot with the process's isolation (the jailed's processes cannot see or signal the host's processes), the network's isolation (the jailed's processes have their own's IP addresses), and the user's isolation (the jailed's root is the unprivileged's user on the host). The jails were the first's complete's container's implementation, and the Solaris's Zones (the 2004's) and the Linux's OpenVZ and Linux-VServer were the parallels.
• The Solaris's Zones (the 2004). The Solaris's Zones: the global's zone (the host) and the non-global's zones (the containers), with the isolated's views (the file systems, the processes, the network, the users) and the resource's limits (the CPU's, the memory's, the network's). The Zones were the enterprise's container's implementation, and their design influenced the modern's containers.
• The Linux's namespaces and cgroups (the 2002–2008). The Linux's kernel's additions: the namespaces (the mount's in the 2002's, the UTS's, the IPC's, the PID's, the network's, the user's in the 2006–2013's) and the cgroups (the process's groups' in the 2006–2008's). The Linux's kernel's mechanisms were the containers' foundation, and the observation is the reason the Linux's containers were the modern's containers' platform.
• The LXC (the 2008). The LXC (the Linux Containers): the user-space's tooling (the lxc-*'s commands) that combined the namespaces, the cgroups, and the chroot into the container's management's interface. The LXC's design was the complete's container's tooling, and it was the Docker's foundation.
• The Docker (the 2013). The Docker: the user-friendly's tooling (the docker's CLI, the Dockerfile's build's format, the Docker Hub's registry) that made the containers the accessible to the developers. The Docker's innovations: the image's concept (the layered's file system: the application and its dependencies), the registry's concept (the images' distribution), the Dockerfile's build's format (the declarative's image's definition), and the single's process's container's model (the container runs the single's application). The Docker's explosion was the containers' accessibility's driver, and the observation is the reason the Docker's name is the container's synonym.
• The OCI (the 2015). The Open Container Initiative: the standards (the Runtime's Specification, the Image's Specification, the Distribution's Specification) that made the containers the interoperable (the same's image runs on the different's runtimes, the same's runtime runs the different's images). The OCI's standards were the containers' ecosystem's foundation, and the observation is the reason the OCI's specifications are the containers' authority.
• The Kubernetes (the 2014). The Kubernetes (the Google's, the Borg's successor): the orchestration's platform (the containers' scheduling, the scaling, the self-healing, the service's discovery, the configuration's management) that made the containers the production's workloads' platform. The Kubernetes's design was the containers' scalability's driver, and the observation is the reason the Kubernetes is the containers' orchestration's standard.

4.3 The containers' mechanisms

The containers' mechanisms are the Linux's kernel's features that the runtimes combine (the Lesson 12.6's sandboxing's discussion):

The namespaces (the isolated's views)

The eight namespaces (the mount, the PID, the network, the user, the IPC, the UTS, the cgroup, the time) give the container's processes the isolated's views of the system's resources. The container's runtime (the runc, the crun) creates the namespaces (the clone's and the unshare's system calls), and the container's processes see only the container's resources. The observation: the namespaces are the views, not the limits, and the cgroups are the complementary's mechanism (the Lesson 12.6's discussion).

The cgroups (the limited's usage)

The cgroups (the v1's and the v2's) limit the container's processes' resource's usage: the CPU's shares and quotas (the cpu.weight's and the cpu.max's), the memory's limits (the memory.max's and the memory.high's), the I/O's weights and limits (the io.weight's and the io.max's), and the processes' count (the pids.max's). The container's runtime places the container's processes in the container's cgroup (the systemd's machine.slice's or the libpod's or the docker's hierarchy), and the cgroup's controllers enforce the limits.

The union file systems (the packaged's file system)

The union file systems (the overlayfs, the aufs, the devicemapper, the btrfs, the zfs) present the multiple's file system's layers as the single's merged's view. The container's image is the stack of the read-only's layers (the base's image, the application's dependencies, the application's code), and the container's writes go to the writable's layer (the overlayfs's upper's directory). The union file system's advantages: the storage's efficiency (the layers are the shared between the containers: the same's base's image is the stored the once), the fast's startup (the layers are the mounted, not the copied), and the container's isolation (the writes are the contained). The overlayfs is the modern's default (the Linux 3.18's, the production's stable in the Linux 4.x).

The capabilities (the restricted's privileges)

The Linux's capabilities (the Lesson 12.5's discussion) split the root's privileges, and the container's runtime drops the unnecessary's capabilities and keeps the specific's. The Docker's default: the dropped's set (the CAP_SYS_ADMIN, the CAP_SYS_PTRACE, the CAP_SYS_MODULE, the CAP_NET_ADMIN, and the others: the dangerous's capabilities) and the kept's set (the CAP_CHOWN, the CAP_DAC_OVERRIDE, the CAP_FOWNER, the CAP_KILL, the CAP_NET_BIND_SERVICE, the CAP_NET_RAW, the CAP_SETGID, the CAP_SETUID, and the others: the ~14's default's set). The --cap-drop=ALL --cap-add=CAP_*'s options let the administrators' finer's control.

The seccomp's filter (the system calls' restrictions)

The seccomp's filter (the Lesson 12.6's discussion): the container's runtime installs the default's filter (the Docker's and the Podman's default's profiles: the ~44's blocked's system calls, including the mount, the reboot, the kexec_load, the init_module, the ptrace), so the container's processes' system calls are the restricted's set. The seccomp's filter is the containers' security's central's mechanism (the attack's surface's reduction: the kernel's vulnerabilities' exploitation is the harder).

The LSM's profiles (the mandatory access control)

The Linux's Security Modules (the SELinux's and the AppArmor's: the Lesson 12.5's discussion) confine the container's processes: the SELinux's container_t's type (the container's processes' label) and the container_file_t's type (the container's files' label), or the AppArmor's docker-default's profile (the container's processes' confinement). The LSM's profiles are the containers' security's additional's layer, and the observation is the reason the containers' security's configurations include the LSM's settings.

The user's namespace and the rootless's containers

The user's namespace (the CLONE_NEWUSER) maps the container's root (the UID 0 inside) to the unprivileged's user (the UID 1000's mapping) on the host, so the container's root's privileges are the bounded (the host's kernel sees the unprivileged's user), and the container's escape gives the attacker only the unprivileged's user's privileges. The rootless's containers (the Podman's default) are the security's recommended's design for the unprivileged's users' workloads.

The no-new-privileges's setting

The no_new_privs's bit (the prctl's PR_SET_NO_NEW_PRIVS) prevents the setuid's and the setgid's programs and the file's capabilities from granting the additional's privileges, so the container's processes cannot escalate their privileges through the setuid's binaries. The Docker's and the Podman's --security-opt no-new-privileges's option sets the bit.

4.4 The OCI standards

The Open Container Initiative (the OCI) is the open-source's project (the Linux Foundation's, the 2015's) that defines the containers' standards. The OCI's specifications:

The Runtime's Specification

The specification of the container's runtime's interface: the container's configuration (the config.json's file: the processes, the root's file system, the mounts, the namespaces, the cgroups, the capabilities, the seccomp's filters, the hooks), the lifecycle (the create, the start, the kill, the delete, the state), and the runc's and the crun's implementations. The specification's purpose: the runtimes' interoperability (the same's config.json runs on the different's runtimes).

The Image's Specification

The specification of the container's image's format: the manifest (the manifest.json's file: the layers' list, the config's file, the digests), the config (the config.json's file: the environment's variables, the entrypoint, the command, the exposed's ports, the volumes, the labels), the layers (the tar's archives: the file system's changes), and the media's types (the application/vnd.oci.image.layer.v1.tar+gzip's and the others). The specification's purpose: the images' interoperability (the same's image runs on the different's runtimes and the different's registries).

The Distribution's Specification

The specification of the containers' distribution: the registry's API (the /v2/'s endpoints: the _catalog, the <name>/tags/list, the <name>/manifests/<reference>, the <name>/blobs/<digest>), the pull's and the push's flows, and the authentication's schemes. The specification's purpose: the registries' interoperability (the same's image is pushed to and pulled from the different's registries: the Docker Hub, the Quay, the GitHub's Container Registry, the Harbor, the cloud's providers').

The OCI's standards' design's observations:

• The interoperability's value. The OCI's standards made the containers the interoperable: the same's image runs on the different's runtimes (the runc, the crun, the containerd, the CRI-O), the different's engines (the Docker, the Podman, the Kubernetes's kubelet), and the different's registries. The observation: the interoperability is the containers' ecosystem's foundation, and the observation is the reason the OCI's specifications are the containers' authority.
• The standards' separation of the concerns. The Runtime's Specification separates the runtime (the container's creation and the execution) from the engine (the image's management, the networking, the storage, the orchestration), and the Image's Specification separates the image's format from the image's content. The observation: the separation of the concerns enabled the ecosystem's diversity (the many's runtimes, the many's engines, the many's registries).

4.5 The container's runtimes

The container's runtimes are the software that creates and runs the containers (the OCI's Runtime's Specification's implementations):

• The low-level's runtimes (the runc, the crun). The low-level's runtimes create the container (the namespaces, the cgroups, the root's file system, the capabilities, the seccomp's filters) from the OCI's configuration and the image's layers. The runc (the Go's implementation: the Docker's and the containerd's default) and the crun (the C's implementation: the Podman's default, the faster and the smaller) are the classical's implementations, and the observation: the low-level's runtimes are the short-lived (they create the container and exit, and the container's processes run on the host's kernel).
• The high-level's runtimes (the containerd, the CRI-O). The high-level's runtimes manage the container's lifecycle (the image's pulling, the container's creation, the container's start, the container's stop, the image's snapshots, the network's management) and delegate the container's creation to the low-level's runtimes (the runc, the crun). The containerd (the Docker's contribution: the containerd's project) and the CRI-O (the Red Hat's contribution: the Kubernetes's CRI's implementation) are the classical's implementations.
• The CRI (the Container Runtime Interface). The Kubernetes's interface (the CRI) that the high-level's runtimes implement, so the Kubernetes's kubelet can use the different's runtimes (the containerd, the CRI-O, the Docker's via the dockershim's compatibility's layer, the deprecated in the Kubernetes 1.24's). The observation: the CRI is the Kubernetes's and the runtimes' interface's standard, and the observation is the reason the Kubernetes's clusters can use the different's runtimes.

4.6 The container's images

The container's image is the packaged's file system: the application and its dependencies, the configuration, and the metadata. The image's structure:

• The layers. The image is the stack of the read-only's layers: each layer is the file system's changes (the added, the modified, the deleted files) relative to the previous's layer. The typical's image: the base's layer (the operating system's file system: the ubuntu:22.04's, the alpine:3.18's, the debian:12's), the dependencies' layers (the runtimes: the Python's, the Node.js's, the Java's), and the application's layer (the application's code). The layers' design's advantages: the storage's efficiency (the layers are the shared between the images: the same's base's layer is the stored the once), the fast's distribution (the layers are the cached and the transferred the incrementally), and the reproducibility (the image's digest identifies the exact's contents).
• The overlayfs's mounting. The container's runtime mounts the image's layers with the union file system (the overlayfs's lower's directories: the read-only's layers; the upper's directory: the writable's layer; the work's directory: the overlayfs's internal's work; the merged's directory: the container's view). The container's writes go to the upper's directory (the writable's layer), and the container's reads see the merged's view. The observation: the union file system's design is the containers' storage's efficiency's and the isolation's mechanism.
• The registries. The registries are the images' repositories: the Docker Hub (the public's), the Quay.io, the GitHub's Container Registry, the GitLab's Container Registry, the AWS's ECR, the Azure's ACR, the Google's Artifact Registry, and the enterprises' private's registries (the Harbor, the Nexus, the Artifactory). The registries' functions: the images' storage, the images' distribution (the pull's and the push's), the images' security's scanning (the vulnerabilities' detection), the images' signing (the content's trust: the Notary's, the Cosign's), and the access's control.
• The image's naming and the digests. The image's reference: the registry/namespace/name:tag's format (the docker.io/library/ubuntu:22.04's, the ghcr.io/myorg/myapp:v1.0.0's), and the image's digest (the sha256:...'s: the content-addressable's identifier: the image's exact's contents). The observation: the tags are the mutable (the same's tag can point to the different's digests) and the digests are the immutable (the exact's contents), and the observation is the reason the production's deployments should use the digests, not the tags.

4.7 The container's lifecycle

The container's lifecycle's stages:

1. The image's pulling. The engine (the Docker, the Podman, the containerd) pulls the image from the registry (the layers are the downloaded and the cached), and the image is the stored locally.
2. The container's creation. The engine creates the container (the create's operation): the runtime (the runc, the crun) sets up the container's environment (the namespaces, the cgroups, the root's file system (the overlayfs's mount), the capabilities, the seccomp's filters, the LSM's profiles), and the container's state is the created.
3. The container's start. The engine starts the container (the start's operation): the container's process (the entrypoint's command) runs, and the container's state is the running.
4. The container's execution. The container's processes run, and the container's state includes the running's processes, the resource's usage (the cgroups' metrics), the file system's changes (the writable's layer), and the network's connections.
5. The container's pause and the resume. The engine can pause the container (the pause's operation: the cgroup's freezer's controller suspends the processes) and resume it (the unpause's operation). The pause's use's cases: the snapshots, the checkpoint and the restore (the CRIU's), the debugging.
6. The container's stop. The engine stops the container (the stop's operation: the SIGTERM's signal, the grace's period, the SIGKILL's signal), and the container's state is the exited. The container's exit's code and the logs are the recorded.
7. The container's removal. The engine removes the container (the rm's operation): the writable's layer is the deleted, the container's resources (the cgroups, the namespaces, the network's interfaces) are the released, and the container's state is the removed. The observation: the container's removal does not affect the image (the layers are the shared), and the observation is the reason the containers are the ephemeral (the stateless's design: the state is the volumes' or the external's services').

4.8 The container's security

The container's security is the layered's design (the Lesson 12.6's sandboxing's discussion applied to the containers), and its considerations:

• The isolation's strength. The containers' isolation is the kernel's mechanisms' (the namespaces and the cgroups), and the containers' security depends on the kernel's security: the kernel's vulnerability (the Dirty COW, the Dirty Pipe, the OverlayFS's bugs, the io_uring's bugs) can be the container's escape's vector. The observation: the containers' shared's kernel is the VMs' separate's kernel's security's difference, and the observation is the reason the high-security's multi-tenancy uses the VMs (or the VM's and the container's combination: the Kata Containers, the gVisor, the Firecracker).
• The attack surface. The containers' attack surface: the container's processes (the application's vulnerabilities: the SQL's injection, the command's injection, the deserialization), the container's image (the base's image's vulnerabilities, the dependencies' vulnerabilities, the secrets' leakage), the runtime (the runc's, the crun's, the containerd's vulnerabilities: the CVE-2019-5736's), the engine (the Docker's, the Podman's), the orchestration (the Kubernetes's RBAC, the network's policies), and the host's kernel.
• The container's escapes. The container's escape is the attacker's exploit of the kernel's, the runtime's, or the configuration's vulnerability to break the isolation: the classical's examples: the CVE-2019-5736 (the runc's: the container's process overwrites the host's runc binary), the CVE-2020-15257 (the containerd's: the abstract's Unix's socket's exposure), the CVE-2021-25741 (the Kubernetes's subpath's: the volumeMounts's subPath's traversal), the CVE-2022-0185 (the Linux's kernel's legacy_parse_param's heap's overflow), the CVE-2022-0492 (the Linux's kernel's cgroups's release_agent's), and the CVE-2022-0847 (the Dirty Pipe's). The escapes' mitigations: the timely's patching (the kernel's, the runtime's, the engine's), the minimal's configurations (the dropped's capabilities, the restrictive's seccomp's profiles, the read-only's file systems), the rootless's containers, the user's namespaces, the LSM's profiles, the monitoring (the runtime's security's tools: the Falco, the Sysdig, the Tetragon), and the VM's and the container's combination (the Kata Containers, the gVisor, the Firecracker).
• The security's configurations. The containers' security's best's practices: the non-root's user (the USER's directive in the Dockerfile, the --user's option), the read-only's root's file system (the --read-only's option with the tmpfs's mounts for the writable's paths), the dropped's capabilities (the --cap-drop=ALL and the --cap-add='s specific's capabilities), the seccomp's profiles (the default's or the custom's), the LSM's profiles, the no-new-privileges's setting, the minimal's images (the distroless's, the alpine's, the scratch's: the smaller's attack surface), the images' scanning (the Trivy's, the Clair's, the Grype's), the secrets' management (the Kubernetes's secrets, the HashiCorp's Vault, the external's secret's stores: the sealed-secrets's, the external-secrets's), and the runtime's security's tools (the Falco's, the Sysdig's, the Tetragon's).

4.9 The containers' use's cases

The containers' use's cases:

• The microservices. The containers are the microservices' architecture's natural's unit: the each service is the container (the single's responsibility, the independent's deployment, the independent's scaling), and the observation is the reason the microservices and the containers are the synonyms in the modern's architectures.
• The CI/CD's pipelines. The containers are the CI/CD's pipelines' natural's unit: the build's environments (the consistent's toolchains), the test's environments (the isolated's testing, the reproducible's configurations), and the deployment's artifacts (the images: the same's artifact from the build to the production). The observation: the containers' consistency (the "it works on my machine" problem's solution) is the CI/CD's pipelines' central's benefit.
• The serverless's platforms. The containers are the serverless's platforms' unit (the AWS's Lambda, the Google's Cloud Functions, the Azure's Functions, the Cloudflare's Workers, the Knative's, the OpenFaaS's): the platform's runtime runs the function as the container (the microVM's or the container's isolation), and the observation: the serverless's platforms hide the containers' management but the containers are the underlying's unit.
• The cloud's native's applications. The containers are the cloud's native's applications' unit (the Kubernetes's clusters, the managed's services: the AWS's ECS and EKS, the Google's GKE, the Azure's AKS, the Red Hat's OpenShift): the applications are the containers, and the platforms manage the deployment, the scaling, the networking, the storage, and the security.
• The development and the testing. The containers are the development's and the testing's environments: the developers run the containers (the consistent's environments: the same's image on the laptop and the CI's server), the testers run the containers (the isolated's tests, the reproducible's results), and the trainers run the containers (the pre-configured's environments).
• The batch's jobs and the data's processing. The containers are the batch's jobs' and the data's processing's unit (the ML's training's jobs, the ETL's pipelines, the scheduled's tasks): the containers package the jobs' dependencies, and the platforms (the Kubernetes's jobs and the CronJobs, the AWS's Batch, the Google's Cloud Run's jobs) manage the execution.

4.10 The containers' and the VMs' trade-offs

The containers' and the VMs' trade-offs:

The trade-offs' observations:

• The complementary's character. The containers and the VMs are the complementary, not the alternatives: the containers' advantages (the density, the startup, the resource's usage) and the VMs' advantages (the isolation, the different's OS) are the different's trade-offs, and the modern's systems combine the two (the containers inside the VMs: the Kubernetes's nodes are the VMs that run the containers; the Kata Containers: the containers' interfaces with the microVMs' isolation; the Firecracker's microVMs: the serverless's and the container's workloads).
• The workloads' matching. The workloads' characteristics determine the choice: the microservices (the many's small's services: the containers), the legacy's applications (the old's operating systems: the VMs), the multi-tenant's environments (the untrusted's code: the VMs' or the microVMs' isolation), the CI/CD's pipelines (the containers), the different's OS's requirements (the VMs). The observation: the choice depends on the workload's requirements (the isolation's strength, the operating system, the density, the startup's time), and the modern's platforms support the both.
• The security's considerations. The multi-tenant's environments (the untrusted's code, the different's customers) require the stronger's isolation (the VMs or the microVMs), and the single-tenant's environments (the trusted's code, the same's organization) can use the containers. The observation: the security's considerations are the multi-tenant's environments' decisive's factor, and the observation is the reason the cloud's providers offer the both (the VMs: the EC2's instances, the GCE's VMs; the containers: the ECS's, the EKS's, the GKE's, the AKS's services).

4.11 The containers' trends and the future

The containers' trends and the future:

• The container's and the VM's convergence. The modern's systems blur the containers' and the VMs' boundaries: the Kata Containers (the containers' interfaces with the microVMs' isolation: the OCI's runtime with the VM's boundary), the gVisor (the user-space's kernel: the application's kernel's emulation in the user-space), the Firecracker (the microVMs: the serverless's and the container's workloads), the KubeVirt (the VMs' management in the Kubernetes's clusters), the libkrun's and the crun's VM's support, and the WebAssembly's sandboxes (the wasmtime's, the wasmedge's: the portable's binaries with the sandbox's isolation).
• The rootless's containers and the user's namespaces. The rootless's containers (the Podman's default, the Docker's rootless's mode) use the user's namespace to run the containers as the unprivileged's users, and the observation: the rootless's containers are the security's recommended's design for the unprivileged's users' workloads (the container's root is the bounded, and the container's escape gives the attacker only the unprivileged's user's privileges).
• The WebAssembly's containers. The WebAssembly (the WASM) is the portable's binary's format with the sandbox's isolation, and the WASM's runtimes (the wasmtime, the wasmer, the wasmedge, the wasmcloud) provide the container's alternatives: the portable's binaries (the cross-platform, the cross-architecture), the fast's startup (the microseconds), the low's memory's overhead (the KBs), and the strong's sandbox's isolation. The observation: the WASM's containers are the emerging's alternative for the specific's workloads (the edge's computing, the serverless's functions, the plugin's systems), and the observation is the reason the WASM's ecosystem is the active's area.
• The confidential containers. The confidential computing's mechanisms (the AMD's SEV, the Intel's TDX, the ARM's CCA) and the containers: the confidential's containers (the enclave-cc's, the occlum's, the gramine's) protect the containers' memory from the host's operators, and the observation: the confidential's containers are the regulated's workloads' (the healthcare, the finance, the government) and the multi-tenant's clouds' requirements.
• The serverless's containers. The serverless's platforms (the AWS's Lambda, the Google's Cloud Run, the Azure's Container Apps, the Cloudflare's Workers, the Knative's) run the containers as the functions, and the observation: the serverless's containers are the containers' abstraction's extreme's case (the developers package the containers, and the platforms manage the everything: the deployment, the scaling, the networking, the security), and the observation is the reason the serverless's platforms are the modern's computing's central's trend.
• The supply chain's security. The containers' supply chain's security: the images' signing (the Notary's, the Cosign's, the Sigstore's), the images' attestation (the SLSA's, the in-toto's), the SBOM's generation (the Software's Bill's of Materials: the Syft's, the Trivy's, the SPDX's), the vulnerabilities' scanning (the Trivy's, the Grype's, the Clair's), and the admission's controllers (the Kubernetes's OPA's Gatekeeper, the Kyverno's, the Sigstore's policy's controller). The observation: the supply chain's security is the containers' security's emerging's concern, and the observation is the reason the modern's platforms include the supply chain's security's tools.

4.12 Common misconceptions

05 Worked example

The following example traces a container's creation and execution through the container's mechanisms, inspects the container's resources, and demonstrates the container's and the VM's comparison.

# Check the container's tools
which podman runc crun 2>/dev/null
podman --version 2>/dev/null
podman info 2>/dev/null | head -30

# Pull an image
podman pull docker.io/library/alpine:3.18 2>/dev/null

# List the images
podman images 2>/dev/null

# Inspect the image's layers
podman inspect docker.io/library/alpine:3.18 2>/dev/null | \
    python3 -c "import sys, json; data = json.load(sys.stdin); print(json.dumps(data[0]['RootFS'], indent=2))"

# Create and run a container
podman run -d --name demo-container \
    --memory 128m --cpus 0.5 \
    --security-opt no-new-privileges \
    docker.io/library/alpine:3.18 \
    sleep 3600

# Inspect the container
podman inspect demo-container 2>/dev/null | head -60

# The container's state
podman ps 2>/dev/null
podman stats --no-stream demo-container 2>/dev/null

# The container's processes
podman top demo-container 2>/dev/null

# The container's namespaces
CONTAINER_PID=$(podman inspect -f '{{.State.Pid}}' demo-container 2>/dev/null)
ls -la /proc/$CONTAINER_PID/ns/

# The container's cgroup's limits
cat /sys/fs/cgroup/machine.slice/libpod-*.scope/memory.max 2>/dev/null
cat /sys/fs/cgroup/machine.slice/libpod-*.scope/cpu.max 2>/dev/null

# The container's capabilities
sudo nsenter -t $CONTAINER_PID -a capsh --print 2>/dev/null | head -10

# The container's seccomp's status
sudo grep Seccomp /proc/$CONTAINER_PID/status 2>/dev/null

# The container's file system
podman exec demo-container ls -la /
podman exec demo-container mount | head -10

# The container's logs
podman logs demo-container 2>/dev/null

# The container's cleanup
podman rm -f demo-container 2>/dev/null

# The container's mounts
podman exec demo-container mount | grep overlay

# The overlayfs's layers
CONTAINER_PID=$(podman inspect -f '{{.State.Pid}}' demo-container 2>/dev/null)
sudo cat /proc/$CONTAINER_PID/mountinfo 2>/dev/null | grep overlay

# The overlayfs's directories
sudo find /var/lib/containers/storage/overlay/ -maxdepth 2 -type d 2>/dev/null | head -20

# The container's writable's layer
podman exec demo-container touch /test-file
sudo find /var/lib/containers/storage/overlay/ -name 'test-file' 2>/dev/null

# The container's image's layers
podman image tree docker.io/library/alpine:3.18 2>/dev/null | head -20

# The container's file system's changes
podman diff demo-container 2>/dev/null

# The container's commit (the new image from the changes)
podman commit demo-container my-alpine-modified 2>/dev/null
podman images | grep my-alpine

# The runc's version
runc --version 2>/dev/null
crun --version 2>/dev/null

# The runtime's binary's size
ls -lh $(which runc) $(which crun) 2>/dev/null

# The runtime's creation of a container (the OCI's bundle)
mkdir -p /tmp/oci-bundle/rootfs
cd /tmp/oci-bundle
skopeo copy docker://docker.io/library/alpine:3.18 \
    oci:/tmp/oci-image:latest 2>/dev/null
skopeo copy oci:/tmp/oci-image:latest \
    dir:/tmp/oci-bundle/rootfs 2>/dev/null
runc spec 2>/dev/null
cat config.json | python3 -m json.tool | head -30

# The runc's container's creation
sudo runc run demo-oci 2>/dev/null &
sleep 2
sudo runc list 2>/dev/null
sudo runc kill demo-oci KILL 2>/dev/null

# Clean up
rm -rf /tmp/oci-bundle /tmp/oci-image 2>/dev/null

# The container's security's options
podman inspect demo-container 2>/dev/null | \
    python3 -c "
import sys, json
data = json.load(sys.stdin)
c = data[0]
print('The capabilities:', c['EffectiveCaps'])
print('The seccomp:', c['HostConfig']['SecurityOpt'])
print('The user:', c['Config']['User'])
print('The read-only:', c['HostConfig']['ReadonlyRootfs'])
print('The privileged:', c['HostConfig']['Privileged'])
"

# The container's user's namespace's mapping
sudo cat /proc/$CONTAINER_PID/uid_map 2>/dev/null
sudo cat /proc/$CONTAINER_PID/gid_map 2>/dev/null

# The container's LSM's labels
sudo cat /proc/$CONTAINER_PID/attr/current 2>/dev/null

# The container's seccomp's filters
sudo cat /proc/$CONTAINER_PID/status | grep Seccomp

# The container's security's tools
which falco sysdig tetragon 2>/dev/null
which trivy grype syft 2>/dev/null

# The container's image's scanning (the demonstration)
trivy image docker.io/library/alpine:3.18 2>/dev/null | head -30

06 Interactive activity

07 Knowledge check

08 Unit discussion

09 Practice exercise

10 Summary

• A container is the isolated's environment in which one or more processes run, with the isolated's view of the system's resources, the limited's resource's usage, and the packaged's file system. The container's and the VM's distinction: the containers run on the shared host's kernel (the kernel's mechanisms: the namespaces and the cgroups), and the VMs run on the separate guest's kernels (the hypervisor and the hardware's virtualization).
• The containers' benefits: the lightweight (the fast's startup, the low's memory's overhead, the high's density), the portability (the images), the consistency (the same's image runs the same's way), the agility (the fast's deployment), the efficiency (the resource's usage), and the ecosystem. The containers' trade-offs: the shared's kernel (the weaker's isolation), the same's kernel's requirement (the different's OS require the VMs), and the complexity (the orchestration, the networking, the storage, the security).
• The containers' history: the chroot (the 1979's), the BSD's jails (the 2000's), the Solaris's Zones (the 2004's), the Linux's namespaces and cgroups (the 2002–2008's), the Linux-VServer and the OpenVZ (the 2000s), the LXC (the 2008's), the Docker (the 2013's), the OCI (the 2015's), and the Kubernetes (the 2014's).
• The containers' mechanisms: the namespaces (the isolated's views), the cgroups (the limited's usage), the union file systems (the packaged's file system: the image's layers and the writable's layer), the capabilities (the restricted's privileges), the seccomp (the system calls' restrictions), the LSM (the mandatory access control), and the user's namespace (the rootless's containers).
• The OCI standards: the Runtime's Specification (the container's runtime's interface), the Image's Specification (the image's format), and the Distribution's Specification (the registry's API). The runtimes: the low-level's (the runc, the crun) and the high-level's (the containerd, the CRI-O), with the CRI (the Kubernetes's Container Runtime Interface).
• The container's image: the layers (the base's, the dependencies', the application's), the overlayfs's mounting (the lower's read-only's layers, the upper's writable's layer), and the registries (the Docker Hub, the Quay, the GitHub's Container Registry, the cloud's providers'). The container's lifecycle: the image's pulling, the container's creation, the start, the execution, the pause, the stop, and the removal.
• The container's security: the isolation's strength (the shared's kernel), the attack surface (the processes, the image, the runtime, the engine, the orchestration, the host's kernel), the escapes (the kernel's, the runtime's, the configuration's vulnerabilities), and the configurations (the non-root's user, the read-only's file system, the dropped's capabilities, the seccomp's profiles, the LSM's profiles, the minimal's images, the scanning, the secrets' management).
• The containers' use's cases: the microservices, the CI/CD's pipelines, the serverless's platforms, the cloud's native's applications, the development and the testing, and the batch's jobs.
• The containers' and the VMs' trade-offs: the containers' lightweight (the density, the startup, the resource's usage) versus the VMs' isolation (the separate's kernels, the different's OS); the two are the complementary, and the modern's systems combine them.
• The containers' trends: the container's and the VM's convergence (the Kata Containers, the gVisor, the Firecracker, the KubeVirt), the rootless's containers, the WebAssembly's containers, the confidential containers, the serverless's containers, and the supply chain's security.

11 References

• Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 18.
• Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, the Virtualization chapters. Available online at ostep.org.
• Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 7.
• Kerrisk, M. The Linux Programming Interface. No Starch Press, 2010, and the man 7 namespaces, the man 7 cgroups, and the man 2 clone pages.
• Rosen, R. Resource Management: The Linux and the SUSE Approaches.
• Merkel, D. Docker: Lightweight Linux Containers for Consistent Development and Deployment. Linux Journal, 2014.
• Bernstein, D. Containers and Cloud: From LXC to Docker to Kubernetes. IEEE Cloud Computing, 2014.
• Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. Borg, Omega, and Kubernetes. ACM Queue, 2016.
• The Open Container Initiative (OCI)'s specifications: the Runtime's Specification, the Image's Specification, and the Distribution's Specification.
• Rice, L. Container Security. O'Reilly, 2020.
• NIST SP 800-190. Application Container Security Guide. National Institute of Standards and Technology.
• Linux man-pages: man 7 namespaces, man 2 clone, man 2 unshare, man 1 unshare, man 1 nsenter, man 7 cgroups, man 1 systemd-cgls, man 1 systemd-cgtop, man 1 systemd-run, man 5 systemd.resource-control, man 2 seccomp, man 7 capabilities, man 8 setcap, man 5 overlay, man 8 mount, man 1 podman, man 1 podman-run, man 1 podman-exec, man 1 podman-build, man 1 buildah, man 1 skopeo, man 1 docker, man 1 docker-run, man 1 runc, man 1 crun, man 1 ctr, man 1 crictl, man 1 systemd-nspawn, man 1 machinectl, man 1 lsns, man 1 dmesg, man 5 proc.