Container Orchestration

01 Introduction

Lesson 13.4 presented Docker: the platform that made the containers the accessible (the images, the Dockerfile, the registries, the CLI, the ecosystem). The Docker's design's scope: the single's host's containers (the docker run's command, the Compose's multi-container's applications on the one host). The Docker's design's limits: the single's host's failures (the host's crash stops the containers), the manual's scaling (the operator runs the docker run's commands for the many's containers), the manual's placement (the operator decides which host runs which container), and the manual's networking (the operator configures the containers' connectivity). The orchestration is the answer to the limits: the platforms that manage the containers at scale — across the many's hosts, with the automated's scheduling, the automated's scaling, the automated's healing, and the automated's networking.

The container orchestration is the automated's management of the containers' lifecycle, the placement, the networking, the storage, and the configuration across the clusters of the many's hosts. The orchestration's concerns:

• The scheduling. The orchestrator decides which host runs which container: the scheduler considers the containers' resource's requests (the CPU, the memory), the hosts' available's resources, the constraints (the affinity, the anti-affinity, the taints and the tolerations, the node's selectors), and the policies (the spread, the bin-packing, the topology's awareness).
• The scaling. The orchestrator adjusts the containers' count (the horizontal's scaling: the replicas' count) and the containers' resources (the vertical's scaling: the CPU's and the memory's requests and limits) based on the demand: the metrics (the CPU's usage, the custom's metrics), the schedules (the time-based's scaling), and the events (the queues' lengths, the traffic's rates).
• The self-healing. The orchestrator detects the containers' failures (the crashed's processes, the failed's health's checks), and it restarts or replaces the failed's containers: the desired's state (the replicas' count) is the orchestrator's specification, and the orchestrator's controllers (the reconciliation's loops) make the actual's state match the desired's state.
• The service's discovery. The orchestrator provides the containers' the network's identities (the services' names, the DNS's names, the virtual's IPs), so the containers can find and communicate with the other's containers (the services' names' resolution, the load's balancing, the endpoints' management) without the manual's configuration.
• The rolling's updates. The orchestrator updates the applications without the downtime: the rolling's updates (the containers are the replaced the incrementally: the new's version's containers are the started, the old's version's containers are the stopped), the blue-green's deployments (the two's environments: the new's version is the deployed, the traffic is the switched), the canary's releases (the small's percentage of the traffic to the new's version, the gradual's increase), and the rollbacks (the failed's updates are the reverted).
• The configuration's management. The orchestrator manages the applications' configuration (the ConfigMaps, the environment's variables, the command's arguments) and the secrets (the Secrets, the external's secret's stores), so the containers' configuration is the declared, the versioned, and the injected at the runtime.
• The storage's management. The orchestrator manages the containers' persistent's storage (the volumes, the PersistentVolumes, the PersistentVolumeClaims, the StorageClasses), so the containers' data is the persistent's and the portable's across the hosts.
• The observability. The orchestrator provides the containers' and the applications' observability (the logs, the metrics, the traces, the events), so the operators can monitor the applications' health and diagnose the failures.

The Kubernetes is the orchestration's standard: the open-source's platform (the Google's, the 2014's release) that is the descendant of the Google's internal's Borg (the 2003's) and Omega (the 2010's) systems. The Kubernetes's design: the declarative's model (the operators declare the desired's state, and the Kubernetes's controllers make the actual's state match the desired's state), the extensible's architecture (the API's server, the controllers, the schedulers, the kubelets, the CRDs, the operators), the portable's abstractions (the pods, the services, the deployments, the ConfigMaps, the Secrets), and the ecosystem (the Helm, the Istio, the Prometheus, the ArgoCD, the many's tools). The Kubernetes's adoption: the cloud's providers' managed's services (the Amazon's EKS, the Google's GKE, the Microsoft's AKS, the Alibaba's ACK, the DigitalOcean's DOKS), the enterprises' on-premises's clusters (the Red Hat's OpenShift, the Rancher's RKE, the VMware's Tanzu), and the developers' local's clusters (the Docker Desktop's Kubernetes, the minikube, the kind, the k3s). The observation: the Kubernetes is the orchestration's de facto standard, and the observation is the reason the lesson's focus is the Kubernetes's architecture and its objects.

This lesson presents the container orchestration. It defines the orchestration's concept and its concerns. It presents the Kubernetes's architecture (the control's plane, the worker's nodes, the API's server, the etcd, the scheduler, the controller's manager, the kubelet, the kube-proxy, the CRI, the CNI, the CSI). It presents the Kubernetes's objects (the pods, the Deployments, the ReplicaSets, the Services, the Ingresses, the ConfigMaps, the Secrets, the StatefulSets, the DaemonSets, the Jobs, the CronJobs, the namespaces, the RBAC, the network's policies). It presents the alternatives (the Docker Swarm, the Nomad, the Mesos, the Amazon's ECS, the Azure's AKS, the Google's GKE) and the security's considerations. It presents the operations's concerns (the observability, the cost's management, the upgrades) and the ecosystem (the Helm, the Istio, the Prometheus, the ArgoCD). It closes with the orchestration's design's trade-offs and the lesson's synthesis.

02 Learning objectives

This lesson contributes to U13.5 and U13.6, and to course learning outcomes CLO1, CLO7, and CLO8.

03 Required reading

• Operating System Concepts — Chapter 18, Virtual Machines, the sections on the containers and the operating-system-level virtualization. The standard treatment of the containers' concept and the orchestration's context.
• OSTEP — the Virtualization chapters, particularly the discussion of the containers and the namespaces. OSTEP's treatment of the containers' mechanisms is the foundation for the orchestration's discussion, and it is the basis for the worked example in this lesson.
• Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. — Borg, Omega, and Kubernetes. ACM Queue, 2016. The paper that describes the Google's orchestration's systems and the Kubernetes's design, and it is the source for the Kubernetes's design's rationale.
• Bernstein, D. — Containers and Cloud: From LXC to Docker to Kubernetes. IEEE Cloud Computing, 2014. The survey of the containers' evolution and the Kubernetes's role.
• Hightower, K., Burns, B., & Beda, J. — Kubernetes: Up and Running, 3rd ed. O'Reilly, 2022. The comprehensive's practical's reference for the Kubernetes's architecture, the objects, and the operations.
• Burns, B., Beda, J., & Hightower, K. — Kubernetes: Up and Running (the earlier's editions). The Kubernetes's concepts and the practical's guidance.
• Lukša, M. — Kubernetes in Action, 2nd ed. Manning, 2020. The comprehensive's reference for the Kubernetes's architecture and the objects.
• Kelsey Hightower's Kubernetes the Hard Way. The step-by-step's guide that builds the Kubernetes's cluster from the components, and it is the source for the Kubernetes's architecture's understanding.
• The Kubernetes's documentation (kubernetes.io/docs): the concepts (the architecture, the objects, the controllers), the tasks, the tutorials, the reference (the API, the kubectl), and the best practices. The authoritative reference for the Kubernetes's usage.
• The Cloud Native Computing Foundation (CNCF)'s landscape. The map of the orchestration's ecosystem (the platforms, the runtimes, the networking, the storage, the observability, the security, the CI/CD), and it is the source for the ecosystem's discussion.
• Rice, L. — Container Security. O'Reilly, 2020. The comprehensive's treatment of the container's security's mechanisms, the escapes' attacks, and the Kubernetes's security's considerations.
• NIST SP 800-190. — Application Container Security Guide. National Institute of Standards and Technology. The security's reference for the containers and the orchestration.
• The Linux man pages: man 1 kubectl (or the kubectl's reference), man 1 minikube, man 1 kind, man 1 k3s, man 1 k3d, man 1 kubeadm, man 1 helm, man 1 k9s, man 1 stern, man 1 kubectx, man 1 kubens, man 1 crictl, man 1 ctr, man 1 nerdctl, man 1 podman, man 1 docker.

04 Core content

4.1 What container orchestration is

The container orchestration is the automated's management of the containers' lifecycle, the placement, the networking, the storage, and the configuration across the clusters of the many's hosts. The orchestration's purpose: the containers' management at scale (the many's hosts, the many's containers, the many's applications), with the automated's operations (the scheduling, the scaling, the healing, the networking, the updates) and the declarative's model (the operators declare the desired's state, and the orchestrator makes the actual's state match the desired's state).

The orchestration's concerns (the introduction's list):

The scheduling

The orchestrator's scheduler decides which host runs which container: the scheduler considers the containers' resource's requests (the CPU, the memory), the hosts' available's resources (the capacity, the allocations), the constraints (the node's selectors, the affinity, the anti-affinity, the taints and the tolerations), and the policies (the spread, the bin-packing, the topology's awareness), and it places the containers on the hosts that satisfy the requirements.

The scaling

The orchestrator adjusts the containers' count (the horizontal's scaling: the replicas' count) and the containers' resources (the vertical's scaling: the CPU's and the memory's requests and limits) based on the demand: the metrics (the CPU's usage, the memory's usage, the custom's metrics: the queue's lengths, the requests' rates), the schedules (the time-based's scaling), and the events (the alerts, the traffic's rates). The Kubernetes's mechanisms: the HorizontalPodAutoscaler (the HPA: the replicas' adjustment), the VerticalPodAutoscaler (the VPA: the resources' adjustment), and the ClusterAutoscaler (the nodes' adjustment: the cloud's providers' integration).

The self-healing

The orchestrator detects the containers' failures (the crashed's processes, the failed's health's checks, the nodes' failures), and it restarts or replaces the failed's containers. The Kubernetes's mechanisms: the controllers (the reconciliation's loops: the Deployments, the ReplicaSets, the StatefulSets, the DaemonSets) compare the desired's state (the replicas' count) with the actual's state, and they create, delete, or replace the containers as the state requires. The observation: the self-healing is the declarative's model's consequence (the desired's state is the specification, and the controllers make the actual's state match).

The service's discovery

The orchestrator provides the containers' the network's identities: the services' names, the DNS's names, the virtual's IPs, the load's balancing, and the endpoints' management. The Kubernetes's mechanisms: the Services (the stable's virtual's IPs and the DNS's names: the myapp.default.svc.cluster.local's names), the kube-proxy (the load's balancing: the iptables's or the IPVS's rules, or the eBPF's with the Cilium), and the CoreDNS (the DNS's server: the services' and the pods' names' resolution).

The rolling's updates

The orchestrator updates the applications without the downtime: the rolling's updates (the containers are the replaced the incrementally), the blue-green's deployments (the two's environments, the traffic's switch), the canary's releases (the small's percentage of the traffic, the gradual's increase), and the rollbacks (the failed's updates' reverts). The Kubernetes's mechanisms: the Deployments (the strategy's field: the RollingUpdate's or the Recreate's; the maxSurge's and the maxUnavailable's parameters), the ReplicaSets (the versions' management), and the kubectl rollout's commands (the status, the history, the undo, the restart).

The configuration's management

The orchestrator manages the applications' configuration and the secrets: the ConfigMaps (the non-sensitive's configuration: the configuration's files, the environment's variables, the command's arguments), the Secrets (the sensitive's data: the passwords, the tokens, the certificates), and the external's secret's stores (the HashiCorp's Vault, the cloud's providers' secret's managers: the AWS's Secrets Manager, the Azure's Key Vault, the Google's Secret Manager; the Kubernetes's integrations: the Secrets Store CSI Driver, the External Secrets Operator).

The storage's management

The orchestrator manages the containers' persistent's storage: the volumes (the pods' volumes: the emptyDir's, the hostPath's, the configMap's, the secret's, the persistentVolumeClaim's), the PersistentVolumes (the cluster's storage's resources), the PersistentVolumeClaims (the pods' storage's requests), the StorageClasses (the dynamic's provisioning's templates: the cloud's providers' storage's classes, the CSI's drivers), and the CSI (the Container Storage Interface: the storage's plugins' standard).

The observability

The orchestrator provides the containers' and the applications' observability: the logs (the containers' stdout's and stderr's streams: the kubectl logs's command, the centralized's logging: the Fluentd's, the Fluent Bit's, the Loki's), the metrics (the resource's usage: the Metrics Server, the Prometheus's, the Grafana's), the traces (the distributed's tracing: the Jaeger's, the Tempo's, the OpenTelemetry's), and the events (the Kubernetes's events: the kubectl get events's command, the kubectl describe's output).

4.2 The Kubernetes's architecture

The Kubernetes's architecture is the cluster's design: the control's plane (the cluster's management's components) and the worker's nodes (the containers' hosts). The components:

The control's plane's components

The cluster's management's components:

• The API's server (kube-apiserver). The cluster's central's component: the REST's API's server that the kubectl's commands, the controllers, the kubelets, and the other's components use. The API's server's roles: the authentication (the tokens, the certificates, the OIDC), the authorization (the RBAC, the ABAC, the webhooks), the admission (the validating's and the mutating's webhooks, the policies), the validation, and the persistence (the etcd's storage). The observation: the API's server is the cluster's single's source of the truth, and the observation is the reason the cluster's every's component communicates with the API's server.
• The etcd. The cluster's key-value's store: the distributed's, the consistent's (the Raft's consensus's protocol) store that holds the cluster's state (the objects: the pods, the deployments, the services, the configs, the secrets). The etcd's roles: the state's persistence, the watch's mechanism (the API's server's and the controllers' watches), and the cluster's consistency. The observation: the etcd is the cluster's critical's component (its loss is the cluster's loss), and the observation is the reason the etcd's backup and its security are the cluster's operations's central's concerns.
• The scheduler (kube-scheduler). The component that assigns the pods to the nodes: the scheduler watches the API's server for the unscheduled's pods, and it selects the nodes (the filtering: the predicates: the resources, the affinity, the taints, the volumes; the scoring: the priorities: the spread, the bin-packing, the topology's awareness; the binding: the selected node's assignment). The observation: the scheduler is the cluster's placement's component, and the observation is the reason the scheduler's policies and the pods' resource's requests and the nodes' labels are the cluster's scheduling's concerns.
• The controller's manager (kube-controller-manager). The component that runs the cluster's controllers: the controllers (the reconciliation's loops) watch the API's server's objects, and they make the actual's state match the desired's state. The controllers: the Deployment's controller (the ReplicaSets' management), the ReplicaSet's controller (the pods' management), the StatefulSet's controller, the DaemonSet's controller, the Job's controller, the CronJob's controller, the Node's controller (the nodes' health's monitoring), the Service's controller, the Endpoints's controller, and the others. The observation: the controllers are the Kubernetes's declarative's model's heart, and the observation is the reason the Kubernetes's design is the the controllers' collection rather than the imperative's commands.
• The cloud's controller's manager (cloud-controller-manager). The component that integrates the cluster with the cloud's providers' APIs: the cloud's controllers (the node's controller: the cloud's instances' lifecycle; the route's controller: the cloud's network's routes; the service's controller: the cloud's load's balancers; the volume's controller: the cloud's storage's provisioning). The observation: the cloud's controller's manager is the Kubernetes's cloud's integration's component.

The worker's nodes's components

The containers' hosts's components:

• The kubelet (kubelet). The node's agent: the component that runs on the every's worker's node, and it manages the node's pods (the pods' lifecycle: the pods' creation, the pods' start, the pods' stop, the pods' deletion), the containers' health (the liveness's, the readiness's, the startup's probes), the volumes' mounting, the CRI's interface (the container's runtime's communication), and the node's status's reporting (the API's server's node's status's updates). The observation: the kubelet is the node's Kubernetes's agent, and the observation is the reason the kubelet's design and the CRI's interface are the Kubernetes's node's management's central's concerns.
• The kube-proxy (kube-proxy). The node's network's proxy: the component that implements the Kubernetes's services' load's balancing on the node (the iptables's rules, the IPVS's, or the eBPF's with the Cilium), and it forwards the traffic to the services' endpoints. The observation: the kube-proxy is the Kubernetes's services' networking's component, and the observation is the reason the kube-proxy's modes (the iptables's, the IPVS's, the eBPF's) are the cluster's networking's concerns.
• The container's runtime (the CRI's implementation). The node's container's runtime: the component that creates and runs the containers (the containerd, the CRI-O, the Docker with the dockershim's compatibility's layer, the deprecated in the Kubernetes 1.24's), and it communicates with the kubelet through the CRI (the Container Runtime Interface). The observation: the CRI's interface is the Kubernetes's and the runtimes' standard, and the observation is the reason the Kubernetes's clusters can use the different's runtimes.
• The CNI's plugins. The node's network's plugins: the CNI (the Container Network Interface: the standard for the containers' network's configuration), and the plugins: the Calico's (the BGP's, the eBPF's), the Cilium's (the eBPF's: the security, the observability), the Flannel's (the VXLAN's), the Weave's, the Antrea's, the Kube-router's. The observation: the CNI's plugins implement the Kubernetes's network's model (the every's pod has its own's IP address, and the pods can communicate with the other's pods across the nodes without the NAT's), and the observation is the reason the CNI's plugins are the cluster's networking's central's concern.
• The CSI's drivers. The node's storage's plugins: the CSI (the Container Storage Interface: the standard for the containers' storage's provisioning and mounting), and the drivers: the cloud's providers' (the AWS's EBS's, the Azure's Disk's, the Google's PD's), the on-premises's (the Ceph's, the NetApp's, the Pure's), the NFS's, the local's. The observation: the CSI's drivers are the Kubernetes's storage's management's mechanism, and the observation is the reason the storage's classes and the CSI's drivers are the cluster's storage's concerns.

4.3 The Kubernetes's objects

The Kubernetes's objects are the cluster's state's records: the declarative's model's vocabulary. The operators declare the objects in the manifests (the YAML's or the JSON's files), and the Kubernetes's controllers make the actual's state match the declared's state. The objects:

The pod

The pod is the Kubernetes's smallest's deployable's unit: the group of the one's or the more's containers that share the network's namespace (the same's IP address and the same's port's space), the storage's volumes, and the lifecycle (the containers are the scheduled, the started, and the stopped together). The pod's design: the containers' co-location (the sidecar's pattern: the helper's containers that augment the main's container: the logging's, the proxy's, the configuration's), the shared's resources, and the pod's specification (the containers, the volumes, the resource's requests and limits, the probes, the security's context, the node's selector, the affinity, the tolerations). The observation: the pod is the Kubernetes's atomic's unit (the operators do not manage the containers directly; they manage the pods), and the observation is the reason the pod's design (the co-location, the shared's resources, the lifecycle) is the Kubernetes's abstractions' foundation.

The Deployment

The Deployment is the stateless's applications' controller: the object that manages the ReplicaSets (the versions) and the pods (the replicas). The Deployment's fields: the replicas (the desired's pod's count), the selector (the pods' labels' selector), the template (the pod's specification), the strategy (the RollingUpdate's or the Recreate's; the maxSurge's and the maxUnavailable's parameters), the revision's history (the revisionHistoryLimit's field), and the progress's deadline. The Deployment's design: the rolling's updates (the new's ReplicaSet is the created, the pods are the replaced the incrementally), the rollbacks (the previous's ReplicaSet is the scaled up, the current's ReplicaSet is the scaled down), and the pause and the resume (the kubectl rollout pause's and the kubectl rollout resume's commands). The observation: the Deployment is the Kubernetes's stateless's applications' standard's object, and the observation is the reason the modern's applications use the Deployments for the web's servers, the API's backends, and the stateless's services.

The ReplicaSet

The ReplicaSet is the pods' count's controller: the object that ensures the specified's number of the pods is the running (the replicas' count: the replicas's field), the pods are the selected (the selector's field), and the pods are the created from the template (the template's field). The ReplicaSet's design: the reconciliation's loop (the actual's pods' count's comparison with the desired's count, and the pods' creation or deletion), and the observation: the ReplicaSet is the Deployment's implementation's detail (the operators rarely manage the ReplicaSets directly), and the observation is the reason the modern's practices use the Deployments (the ReplicaSets' management) rather than the ReplicaSets (the pods' management).

The Service

The Service is the stable's network's identity: the object that provides the stable's virtual's IP address and the DNS's name for the group of the pods (the selector's field: the pods' labels), and it load-balances the traffic across the pods (the endpoints). The Service's types: the ClusterIP's (the cluster's internal's virtual's IP: the default), the NodePort's (the every node's port: the external's access through the nodes' IPs), the LoadBalancer's (the cloud's providers' load's balancers: the external's access through the load's balancer's IP), and the ExternalName's (the DNS's CNAME's alias). The Service's design: the endpoints' management (the Endpoints's object: the pods' IPs and the ports), the kube-proxy's load's balancing (the iptables's, the IPVS's, or the eBPF's), and the DNS's resolution (the CoreDNS: the myapp.default.svc.cluster.local's names). The observation: the Service is the Kubernetes's services' discovery's and the load's balancing's object, and the observation is the reason the applications' communication uses the services' names (the stable's identities) rather than the pods' IPs (the ephemeral's addresses).

The Ingress

The Ingress is the external's HTTP and HTTPS's routing: the object that routes the external's traffic to the services (the host's routing: the example.com's, the api.example.com's; the path's routing: the /'s, the /api's; the TLS's termination: the certificates). The Ingress's design: the Ingress's controllers (the Nginx's, the Traefik's, the HAProxy's, the Envoy's, the Istio's) implement the routing, and the observation: the Ingress is the Kubernetes's external's access's object, and the observation is the reason the modern's applications use the Ingress (the HTTP and the HTTPS's routing) with the Ingress's controllers (the external's load's balancers' and the reverse's proxies' implementations).

The ConfigMap and the Secret

The ConfigMap is the non-sensitive's configuration: the object that holds the configuration's data (the key-value's pairs: the configuration's files, the environment's variables, the command's arguments), and it is the mounted into the pods (the volumes' or the environment's variables' injection). The Secret is the sensitive's data: the object that holds the sensitive's data (the passwords, the tokens, the certificates, the keys), and it is the mounted into the pods (the volumes' or the environment's variables' injection) and the stored in the etcd (the base64-encoded: the encryption at rest is the configuration's option). The design's observation: the ConfigMaps and the Secrets are the Kubernetes's configuration's management's objects, and the observation is the reason the applications' configuration is the declared (the manifests), the versioned (the Git's repositories), and the injected (the pods' mounts) rather than the baked into the images.

The StatefulSet

The StatefulSet is the stateful's applications' controller: the object that manages the stateful's applications (the databases, the caches, the message's queues), with the stable's identities (the pods' names: the myapp-0's, the myapp-1's, the myapp-2's; the network's identities: the headless's services' DNS's names), the stable's storage (the PersistentVolumeClaims's templates: the every pod's volume), and the ordered's deployment (the pods are the created and the deleted in the order). The observation: the StatefulSet is the Kubernetes's stateful's applications' object, and the observation is the reason the databases and the other's stateful's workloads use the StatefulSets (the stable's identities and storage) rather than the Deployments (the interchangeable's pods).

The DaemonSet

The DaemonSet is the every node's pod's controller: the object that ensures the every node (or the selected's nodes: the node's selectors, the affinity, the taints and the tolerations) runs the specified's pod. The DaemonSet's use's cases: the logs' collectors (the Fluentd's, the Fluent Bit's), the monitoring's agents (the Prometheus's node's exporter, the Datadog's agent), the network's plugins (the Calico's, the Cilium's node's components), the storage's drivers (the CSI's node's plugins), and the security's agents (the Falco's). The observation: the DaemonSet is the Kubernetes's per-node's workloads' object, and the observation is the reason the cluster's infrastructure's components (the logs, the monitoring, the networking, the storage) use the DaemonSets.

The Job and the CronJob

The Job is the batch's workloads' controller: the object that runs the pods to the completion (the completions's field: the successful's pods' count; the parallelism's field: the concurrent's pods' count; the backoffLimit's field: the retries' count; the activeDeadlineSeconds's field: the timeout). The CronJob is the scheduled's jobs' controller: the object that runs the Jobs on the schedule (the schedule's field: the cron's expression; the concurrencyPolicy's field: the Allow's, the Forbid's, the Replace's policies; the startingDeadlineSeconds's field; the successfulJobsHistoryLimit's and the failedJobsHistoryLimit's fields). The observation: the Jobs and the CronJobs are the Kubernetes's batch's workloads' objects, and the observation is the reason the data's processing (the ETL's pipelines, the ML's training's jobs), the backups, and the maintenance's tasks use the Jobs and the CronJobs.

The namespace

The namespace is the cluster's logical's partition: the object that provides the scope for the names (the the objects' names are the unique within the namespace, not the cluster), the resource's quotas (the ResourceQuota's objects: the CPU's, the memory's, the objects' counts), the limit's ranges (the LimitRange's objects: the default's and the maximum's resource's limits), the network's policies (the NetworkPolicy's objects: the pods' traffic's rules), and the RBAC's scope (the roles' and the role's bindings' scope). The namespaces' use's cases: the environments' separation (the development, the staging, the production), the teams' separation, the tenants' separation. The observation: the namespace is the Kubernetes's multi-tenancy's and the organization's mechanism, and the observation is the reason the namespaces are the cluster's logical's partitions.

The RBAC (the Role-Based Access Control)

The RBAC is the Kubernetes's authorization's mechanism: the objects (the Role's and the ClusterRole's: the permissions' rules: the API's groups, the resources, the verbs; the RoleBinding's and the ClusterRoleBinding's: the subjects' and the roles' bindings: the users, the groups, the service's accounts), and the authorization's model (the additive's: the every's matching's rule's permission is the granted, the no's deny's rules, the the cluster's admin's role is the cluster's superuser). The observation: the RBAC is the Kubernetes's access's control's mechanism, and the observation is the reason the RBAC's design (the least privilege's principle: the the applications' service's accounts are the the specific's permissions, the the users' roles are the the specific's namespaces' scopes) is the cluster's security's central's concern.

The network's policy

The NetworkPolicy is the pods' traffic's rules: the object that specifies the allowed's ingress and egress's traffic (the pod's selectors, the namespace's selectors, the IP's blocks, the ports, the protocols), and the observation: the NetworkPolicy is the Kubernetes's network's segmentation's mechanism, and the observation is the reason the NetworkPolicy's design (the the namespaces' and the pods' isolation's rules) is the cluster's security's central's concern. The observation: the NetworkPolicy's enforcement requires the CNI's plugins' support (the the Calico's, the Cilium's, the Weave's: the the enforcement's plugins; the the Flannel's: the no's enforcement's support).

4.4 The Kubernetes's declarative's model

The Kubernetes's declarative's model is the design's central's idea: the operators declare the desired's state (the manifests: the YAML's or the JSON's files), and the Kubernetes's controllers (the reconciliation's loops) make the actual's state match the desired's state. The model's components:

• The manifests. The declarative's specifications: the YAML's files (the apiVersion's, the kind's, the metadata's, the spec's sections), and the observation: the manifests are the applications' definitions (the infrastructure's as the code), and the observation is the reason the manifests are the versioned in the Git's repositories.
• The kubectl's commands. The Kubernetes's CLI: the imperative's commands (the kubectl run's, the kubectl create's, the kubectl delete's, the kubectl scale's, the kubectl expose's: the ad-hoc's operations) and the declarative's commands (the kubectl apply -f's, the kubectl diff's, the kubectl delete -f's, the kubectl kustomize's: the manifests' operations). The kubectl's commands: the resource's inspection (the get's, the describe's, the logs's, the exec's, the port-forward's, the cp's), the cluster's inspection (the cluster-info's, the get nodes's, the top's), the rollout's management (the rollout's subcommands: the status's, the history's, the undo's, the restart's), the debugging (the debug's, the events's), and the configuration (the config's subcommands: the set-context's, the set-namespace's).
• The reconciliation's loops. The controllers' mechanism: the every's controller watches the API's server's objects (the watch's mechanism: the etcd's events), and it compares the actual's state (the cluster's resources) with the desired's state (the objects' specifications), and it performs the actions (the objects' creation, the objects' deletion, the objects' updates) to make the actual's state match the desired's state. The observation: the reconciliation's loops are the Kubernetes's declarative's model's heart, and the observation is the reason the Kubernetes's design is the the controllers' collection rather than the imperative's commands.
• The GitOps's model. The GitOps's design: the manifests are the versioned in the Git's repositories (the desired's state's single's source of the truth), and the GitOps's tools (the ArgoCD's, the Flux's) watch the repositories and reconcile the clusters' state with the repositories' state (the pull's model: the tools pull the manifests from the Git, the CI/CD's pipelines push the manifests to the Git, the tools reconcile the clusters, the observation: the GitOps's model is the declarative's model's extension, and the observation is the reason the GitOps's tools are the modern's Kubernetes's operations's standard).

The declarative's model's example (the Deployment's manifest):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: production
  labels:
    app: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: ghcr.io/myorg/web:v1.0.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          volumeMounts:
            - name: config
              mountPath: /etc/app
              readOnly: true
            - name: secrets
              mountPath: /etc/secrets
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: web-config
        - name: secrets
          secret:
            secretName: web-secrets
      securityContext:
        fsGroup: 1000
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: web
                topologyKey: kubernetes.io/hostname

The manifest's observation: the spec's section declares the desired's state (the three's replicas, the container's image, the resources' requests and limits, the probes, the security's context, the volumes, the affinity), and the Kubernetes's controllers (the Deployment's, the ReplicaSet's, the scheduler's, the kubelet's) make the actual's state match. The observation: the manifest is the application's definition (the versioned, the reviewable, the reproducible), and the observation is the reason the declarative's model is the Kubernetes's design's central's idea.

4.5 The Kubernetes's scheduling and networking

The Kubernetes's scheduling: the scheduler assigns the pods to the nodes, and its mechanisms:

• The resource's requests and limits. The pods' containers declare the resource's requests (the guaranteed's resources: the scheduler uses the requests to select the nodes) and the limits (the maximum's resources: the kubelet enforces the limits), and the observation: the requests and the limits are the scheduling's and the enforcement's mechanisms (the cgroups' configuration).
• The node's selection's rules. The nodeSelector's field (the simple's label's selector), the nodeName's field (the specific's node), the affinity (the nodeAffinity's: the nodes' labels' selectors; the podAffinity's: the pods' co-location; the podAntiAffinity's: the pods' separation), and the taints and the tolerations (the nodes' taints: the NoSchedule's, the PreferNoSchedule's, the NoExecute's; the pods' tolerations: the the taints' matching's tolerations).
• The scheduler's policies. The filtering (the predicates: the resources' availability, the affinity, the taints, the volumes, the ports), the scoring (the priorities: the spread, the bin-packing, the topology's awareness, the image's locality), and the binding (the selected's node's assignment), and the observation: the scheduler's design is the Kubernetes's placement's mechanism, and the observation is the reason the scheduler's policies (the the cluster's defaults and the the pods' specifications) are the cluster's scheduling's concerns.
• The scheduler's profiles. The scheduler's profiles (the KubeSchedulerConfiguration's object: the profiles: the default-scheduler's, the high-throughput's; the plugins: the NodeResourcesFit's, the PodTopologySpread's, the InterPodAffinity's, the ImageLocality's; the pluginConfig's), and the observation: the scheduler's profiles are the cluster's scheduling's customization's mechanism.

The Kubernetes's networking: the cluster's network's model, and its mechanisms:

• The network's model. The Kubernetes's network's requirements (the CNI's specification): the every pod has its own's IP address (the cluster's unique's IP), the the pods can communicate with the the other's pods on the the same's node and the the different's nodes without the NAT's, the the nodes can communicate with the the pods, the the pods can communicate with the the external's world (the the NAT's for the the egress's traffic), and the observation: the the network's model's requirements are the CNI's plugins' specifications.
• The CNI's plugins. The CNI's plugins: the Calico's (the the BGP's and the the eBPF's datapath's, the the network's policies' enforcement's), the Cilium's (the the eBPF's: the the networking's, the the security's, the the observability's, the the service's mesh's, the the network's policies'), the Flannel's (the the VXLAN's overlay's: the the simple's, the the no's network's policies' enforcement's), the Weave's, the Antrea's (the the Open vSwitch's), the Kube-router's, and the observation: the CNI's plugins are the cluster's networking's and the network's policies' implementations.
• The services' networking. The services' networking: the ClusterIP's (the the virtual's IP: the the kube-proxy's iptables's rules or the the IPVS's virtual's servers, or the the eBPF's with the the Cilium), the NodePort's (the the every node's port: the the external's access through the the nodes' IPs), the LoadBalancer's (the the cloud's providers' load's balancers: the the external's IP: the the cloud's controller's manager's provisioning's), and the observation: the services' networking is the Kubernetes's services' discovery's and the load's balancing's mechanism.
• The DNS's resolution. The CoreDNS's (the cluster's DNS's server: the services' names: the the myapp.default.svc.cluster.local's format: the the <service>.<namespace>.svc.cluster.local's; the the pods' names: the the headless's services'; the the stateful's pods' names; the the search's domains: the the pod's /etc/resolv.conf's file), and the observation: the DNS's resolution is the Kubernetes's services' discovery's mechanism.

4.6 The orchestration's alternatives

The orchestration's alternatives:

The Docker Swarm

The Docker Swarm is the Docker's native's orchestration (the Docker's, the 2014's): the cluster's swarm (the managers and the workers), the services (the multi-container's applications: the replicated's tasks), the overlay's networks (the multi-host's networks), the secrets and the configs, and the stacks (the Compose's files' deployment). The Swarm's advantages: the simplicity (the Docker's CLI and the Compose's files: the the Docker's familiar's interfaces), the the integration (the Docker's ecosystem: the the images, the the Compose, the the secrets), and the the learning's curve (the the the Docker's knowledge's reuse). The Swarm's disadvantages: the the features (the the fewer's than the the Kubernetes's: the the no's auto-scaling's, the the no's the the advanced's scheduling's), the the ecosystem (the the smaller's than the the Kubernetes's), and the the adoption (the the Kubernetes's dominance: the the Swarm's decline), and the observation: the Swarm's design is the the Docker's simplicity's choice, and the observation is the the reason the the small's deployments and the the Docker's developers use the the Swarm.

The HashiCorp's Nomad

The Nomad is the HashiCorp's orchestration (the HashiCorp's, the 2015's): the cluster's servers (the management's components), the clients (the workloads' hosts), the jobs (the workloads' definitions: the the containers, the the virtual machines, the the raw's binaries, the the Java's applications), and the allocations (the jobs' instances). The Nomad's advantages: the simplicity (the the single's binary, the the simple's configuration), the the flexibility (the the multiple's workloads: the the containers, the the VMs, the the binaries), the the integration (the the HashiCorp's stack: the the Consul, the the Vault, the the Terraform), and the the scale (the the millions of the containers: the the HashiCorp's claims). The Nomad's disadvantages: the the ecosystem (the the smaller's than the the Kubernetes's), and the the adoption (the the Kubernetes's dominance), and the observation: the the Nomad's design is the the HashiCorp's ecosystem's integration's choice.

The Apache Mesos

The Mesos is the Apache's cluster's manager (the UC Berkeley's, the 2011's): the the master's nodes (the the management's components), the the agent's nodes (the the workloads' hosts), the the frameworks (the the workload's managers: the the Marathon's for the the containers, the the Chronos's for the the scheduled's tasks, the the Spark's for the the data's processing), and the the offers (the the resources' proposals). The Mesos's advantages: the the scale (the the thousands of the nodes), the the flexibility (the the multiple's frameworks), and the the maturity (the the years of the production's use: the the Twitter's, the the Airbnb's). The Mesos's disadvantages: the the complexity (the the frameworks' management), and the the adoption (the the Kubernetes's dominance: the the Mesos's decline), and the observation: the the Mesos's design is the the data's center's OS's vision (the the frameworks as the the workloads' managers).

The cloud's providers' orchestration's services

The cloud's providers' orchestration's services: the Amazon's ECS (the Elastic Container Service: the the AWS's native's orchestration: the the task's definitions, the the services, the the clusters, the the Fargate's serverless's compute), the Amazon's EKS (the Elastic Kubernetes Service: the the managed's Kubernetes's clusters), the Google's GKE (the Google Kubernetes Engine: the the managed's Kubernetes's clusters, the the Autopilot's mode), the Microsoft's AKS (the Azure Kubernetes Service), the Alibaba's ACK, the DigitalOcean's DOKS, and the observation: the the cloud's providers' services are the the managed's orchestration (the the control's plane's management's outsourcing: the the upgrades, the the scaling, the the monitoring), and the observation is the the reason the the modern's organizations use the the managed's services rather than the the self-managed's clusters.

4.7 The orchestration's security

The orchestration's security: the clusters' security's considerations (the Lesson 12.1's principles and the Lesson 13.3's and the Lesson 13.4's discussions applied to the Kubernetes):

• The cluster's security's layers. The layers: the cloud's infrastructure's security (the the VPC's, the the security's groups, the the IAM's), the the cluster's security (the the API's server's authentication and the the authorization, the the RBAC, the the admission's controllers, the the etcd's encryption), the the node's security (the the OS's hardening, the the kubelet's security, the the container's runtime's security), the the pod's security (the the security's context, the the pod's security's admission, the the seccomp's, the the AppArmor's, the the SELinux's, the the capabilities), and the the application's security (the the container's security's configurations: the the non-root's user, the the read-only's file system, the the dropped's capabilities), and the observation: the the security's layers are the the defence's in the depth's application (the the Lesson 12.1's principle).
• The API's server's security. The API's server's security's mechanisms: the authentication (the the client's certificates, the the bearer's tokens, the the OIDC's, the the webhook's, the the service's accounts' tokens), the authorization (the the RBAC's, the the ABAC's, the the webhook's, the the node's authorizer), the admission's controllers (the the validating's and the the mutating's: the the PodSecurity's, the the ResourceQuota's, the the LimitRanger's, the the NetworkPolicy's, the the custom's webhooks), and the audit's logging (the the API's server's audit's policy: the the requests' recording), and the observation: the the API's server's security is the the cluster's security's the central's component.
• The pod's security. The pod's security's mechanisms: the the security's context (the the runAsUser's, the the runAsNonRoot's, the the readOnlyRootFilesystem's, the the allowPrivilegeEscalation's, the the capabilities' the drop's, the the seccompProfile's, the the appArmorProfile's, the the seLinuxOptions's, the the fsGroup's, the the sysctls's), the the Pod Security's Standards (the the privileged's, the the baseline's, the the restricted's: the the three's policy's levels: the the namespaces' labels: the the pod-security.kubernetes.io/enforce's), the the Pod Security's Admission (the the admission's controller: the the PSS's enforcement: the the three's modes: the the enforce's, the the audit's, the the warn's), and the observation: the the pod's security is the the cluster's security's the application's layer.
• The network's security. The network's security's mechanisms: the the network's policies (the the pods' ingress and the the egress's rules: the the namespace's selectors, the the pod's selectors, the the IP's blocks, the the ports), the the service's mesh's security (the the mTLS's: the the mutual's TLS's, the the authorization's policies, the the traffic's encryption), and the observation: the the network's security is the the cluster's security's the networking's layer, and the observation is the the reason the the network's policies and the the service's mesh are the the cluster's security's central's concerns.
• The supply chain's security. The supply chain's security's mechanisms: the the images' scanning (the the Trivy's, the the Grype's, the the Clair's), the the images' signing (the the Cosign's, the the Notary's: the the Sigstore's), the the images' attestation (the the SLSA's, the the in-toto's), the the SBOM's (the the Software's Bill's of the Materials: the the Syft's, the the Trivy's, the the SPDX's), the the admission's controllers (the the Kyverno's, the the OPA's Gatekeeper's, the the Sigstore's policy's controller: the the images' verification's the admission's policies), and the observation: the the supply chain's security is the the cluster's security's the emerging's concern.
• The runtime's security. The runtime's security's tools: the the Falco's (the the runtime's security: the the system's calls' monitoring, the the rules, the the alerts), the the Tetragon's (the the eBPF's security's observability: the the Kubernetes's awareness), the the Sysdig's, the the Aqua's, the the StackRox's, and the observation: the the runtime's security's tools are the the cluster's security's the operations's mechanisms (the the detection, the the response).

4.8 The orchestration's operations

The orchestration's operations (the Day 2's concerns): the clusters' management after the initial's deployment:

• The observability. The logs (the the kubectl logs's command, the the centralized's logging: the the Fluentd's, the the Fluent Bit's, the the Loki's, the the Elasticsearch's), the metrics (the the Metrics Server: the the kubectl top's command, the the Prometheus's, the the Grafana's, the the VictoriaMetrics's), the traces (the the Jaeger's, the the Tempo's, the the OpenTelemetry's), and the events (the the kubectl get events's, the the kubectl describe's), and the observation: the the observability is the the cluster's operations's the central's concern.
• The cost's management. The cost's management's tools: the the Kubecost's, the the OpenCost's, the the cloud's providers' cost's management's tools (the the AWS's Cost Explorer, the the Azure's Cost Management, the the GCP's Cost Management), the the resource's requests and the the limits' optimization (the the rightsizing's: the the VPA's), and the observation: the the cost's management is the the cluster's operations's the emerging's concern (the the cloud's bills' the significant's component).
• The upgrades. The upgrades' mechanisms: the the cluster's version's upgrades (the the control's plane's, the the nodes': the the rolling's, the the blue-green's), the the API's versions' deprecations (the the migration's guides), the the add-ons' upgrades (the the CNI's, the the CSI's, the the Ingress's, the the operators'), and the observation: the the upgrades are the the cluster's operations's the periodic's concern.
• The backups. The backups' mechanisms: the the etcd's backups (the the critical's: the the cluster's state), the the persistent's volumes' backups (the the Velero's: the the cluster's resources' and the the volumes' backups), the the manifests' backups (the the Git's repositories: the the GitOps's model), and the observation: the the backups are the the cluster's operations's the essential's concern.
• The disaster's recovery. The disaster's recovery's mechanisms: the the multi-cluster's designs (the the active-active's, the the active-passive's), the the cross-region's replication (the the storage's, the the etcd's), the the failover's procedures, and the observation: the the disaster's recovery is the the cluster's operations's the resilience's concern.

4.9 The orchestration's ecosystem

The orchestration's ecosystem (the Cloud Native Computing Foundation's landscape): the tools that extend the Kubernetes:

• The Helm (the package's management). The Helm is the Kubernetes's package's manager (the the charts: the the packages: the the templates, the the default's values, the the dependencies; the the releases: the the installed's instances; the the repositories: the the chart's repositories), and the observation: the the Helm's design is the the Kubernetes's applications' packaging's and the the deployment's mechanism (the the templating's, the the versioning's, the the rollbacks'), and the observation is the the reason the the Helm is the the Kubernetes's ecosystem's the standard's package's manager.
• The Kustomize (the configuration's management). The Kustomize is the Kubernetes's configuration's customization's tool (the the base's manifests and the the overlays: the the environment's specific's customizations: the the patches, the the name's prefixes, the the labels, the the image's tags), and the observation: the the Kustomize's design is the the Kubernetes's manifests' the templating's alternative (the the plain's YAML's the overlays), and the observation is the the reason the the kubectl kustomize's and the the kubectl apply -k's commands are the the Kubernetes's the configuration's management's standard.
• The Istio (the service's mesh). The Istio is the Kubernetes's service's mesh (the the traffic's management, the the security's: the the mTLS's, the the authorization's policies, the the observability's: the the metrics, the the traces, the the logs), and the observation: the the Istio's design is the the Kubernetes's the networking's the advanced's features (the the traffic's routing, the the resilience's: the the retries, the the timeouts, the the circuit's breakers), and the observation is the the reason the the service's mesh (the the Istio's, the the Linkerd's, the the Consul's Connect's) is the the Kubernetes's the microservices' networking's the standard.
• The Prometheus (the monitoring). The Prometheus is the Kubernetes's monitoring (the the metrics' collection: the the pull's model, the the exporters, the the service's monitors, the the pod's monitors; the the PromQL's query's language; the the Alertmanager's the alerts), and the observation: the the Prometheus's design is the the Kubernetes's the observability's the standard (the the CNCF's the graduated's project), and the observation is the the reason the the Grafana's (the the dashboards) and the the Prometheus's (the the metrics) are the the Kubernetes's the monitoring's the standard.
• The ArgoCD and the Flux (the GitOps). The ArgoCD and the Flux are the Kubernetes's the GitOps's tools (the the Git's repositories as the the desired's state's the single's source of the truth, the the reconciliation's: the the tools' the pull's model, the the clusters' the state's the repositories' the state's the match), and the observation: the the GitOps's design is the the Kubernetes's the declarative's model's the extension, and the observation is the the reason the the GitOps's tools are the the Kubernetes's the operations's the modern's standard.
• The Operators (the application's management). The Operators are the Kubernetes's the applications' the operational's knowledge's the encapsulation (the the custom's resources: the the CRDs; the the controllers: the the reconciliation's loops), and the observation: the the Operators' design is the the Kubernetes's the extensibility's the mechanism, and the observation is the the reason the the Operators (the the Prometheus's Operator, the the PostgreSQL's Operator, the the Kafka's Operator, the the Cert-Manager's) are the the Kubernetes's the complex's applications' the deployment's and the the operations's the standard.

4.10 The orchestration's design's trade-offs

The orchestration's design's trade-offs:

4.11 Common misconceptions

05 Worked example

The following example deploys a multi-tier application to a Kubernetes's cluster, demonstrates the declarative's model's reconciliation, and inspects the cluster's state.

# The namespace's manifest
cat > 00-namespace.yaml << 'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: myapp
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
EOF

# The ConfigMap's manifest
cat > 01-configmap.yaml << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
  namespace: myapp
data:
  nginx.conf: |
    server {
      listen 8080;
      location / {
        proxy_pass http://api:3000;
      }
    }
EOF

# The Secret's manifest
cat > 02-secret.yaml << 'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: db-secrets
  namespace: myapp
type: Opaque
stringData:
  username: myapp
  password: s3cr3t-p4ssw0rd
EOF

# The PostgreSQL's StatefulSet's manifest
cat > 03-postgres.yaml << 'EOF'
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
  namespace: myapp
spec:
  serviceName: db
  replicas: 1
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      securityContext:
        fsGroup: 999
        runAsNonRoot: true
        runAsUser: 999
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: db-secrets
                  key: username
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-secrets
                  key: password
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 1Gi
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: db
  namespace: myapp
spec:
  selector:
    app: db
  ports:
    - port: 5432
      targetPort: 5432
  clusterIP: None
EOF

# The API's Deployment's manifest
cat > 04-api.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: node:20-alpine
          command: ["node", "-e", "require('http').createServer((req,res)=>{res.end('Hello from the API!')}).listen(3000)"]
          ports:
            - containerPort: 3000
          env:
            - name: DB_HOST
              value: db
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: db-secrets
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-secrets
                  key: password
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          livenessProbe:
            httpGet: { path: /, port: 3000 }
            initialDelaySeconds: 5
          readinessProbe:
            httpGet: { path: /, port: 3000 }
            initialDelaySeconds: 3
---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: myapp
spec:
  selector:
    app: api
  ports:
    - port: 3000
      targetPort: 3000
EOF

# The web's Deployment's manifest
cat > 05-web.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: myapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 101
        fsGroup: 101
      containers:
        - name: nginx
          image: nginx:alpine
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d/default.conf
              subPath: nginx.conf
              readOnly: true
            - name: cache
              mountPath: /var/cache/nginx
            - name: run
              mountPath: /var/run
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
              add: ["NET_BIND_SERVICE"]
      volumes:
        - name: config
          configMap:
            name: web-config
        - name: cache
          emptyDir: {}
        - name: run
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: myapp
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080
EOF

# The Ingress's manifest
cat > 06-ingress.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: myapp
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80
EOF

# The HPA's manifest
cat > 07-hpa.yaml << 'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: myapp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
EOF

# The NetworkPolicy's manifest
cat > 08-networkpolicy.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: myapp
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-to-api
  namespace: myapp
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: web
      ports:
        - port: 3000
          protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-db
  namespace: myapp
spec:
  podSelector:
    matchLabels:
      app: db
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - port: 5432
          protocol: TCP
EOF

# Apply all the manifests
kubectl apply -f 00-namespace.yaml
kubectl apply -f 01-configmap.yaml -f 02-secret.yaml
kubectl apply -f 03-postgres.yaml
kubectl apply -f 04-api.yaml -f 05-web.yaml
kubectl apply -f 06-ingress.yaml -f 07-hpa.yaml -f 08-networkpolicy.yaml

# Watch the deployment's rollout
kubectl rollout status deployment/api -n myapp
kubectl rollout status deployment/web -n myapp
kubectl rollout status statefulset/db -n myapp

# The pods' state
kubectl get pods -n myapp
kubectl get deployments -n myapp
kubectl get services -n myapp
kubectl get statefulsets -n myapp
kubectl get pvc -n myapp
kubectl get hpa -n myapp
kubectl get networkpolicies -n myapp

# The deployment's details
kubectl describe deployment api -n myapp

# The pods' placement
kubectl get pods -n myapp -o wide

# The self-healing's demonstration: delete a pod
POD=$(kubectl get pods -n myapp -l app=api -o jsonpath='{.items[0].metadata.name}')
kubectl delete pod $POD -n myapp

# The ReplicaSet's controller's reaction (the new pod's creation)
sleep 2
kubectl get pods -n myapp -l app=api

# The scaling's demonstration: scale the deployment
kubectl scale deployment/api -n myapp --replicas=5
kubectl get pods -n myapp -l app=api

# The rolling's update's demonstration: update the image
kubectl set image deployment/api -n myapp api=node:20-alpine
kubectl rollout status deployment/api -n myapp
kubectl rollout history deployment/api -n myapp

# The rollback's demonstration
kubectl rollout undo deployment/api -n myapp
kubectl rollout status deployment/api -n myapp

# The HPA's state
kubectl get hpa api -n myapp
kubectl describe hpa api -n myapp

# The service's endpoints
kubectl get endpoints -n myapp

# The events
kubectl get events -n myapp --sort-by='.lastTimestamp' | tail -20

# The cluster's info
kubectl cluster-info
kubectl version
kubectl config current-context
kubectl config get-contexts

# The nodes
kubectl get nodes
kubectl describe nodes | head -60

# The control's plane's components
kubectl get pods -n kube-system
kubectl get pods -n kube-system -l component=kube-apiserver

# The API's resources
kubectl api-resources | head -40
kubectl api-versions | head -20

# The cluster's capacity
kubectl top nodes
kubectl top pods -n myapp

# The namespaces
kubectl get namespaces

# The RBAC's objects
kubectl get roles,rolebindings -A | head -20
kubectl get clusterroles,clusterrolebindings | head -10
kubectl auth can-i --list -n myapp

# The service's accounts
kubectl get serviceaccounts -n myapp

# The CRDs (if any operators are installed)
kubectl get crds | head -20

# The events
kubectl get events -A --sort-by='.lastTimestamp' | tail -20

# The pod's state and the events
kubectl get pods -n myapp
kubectl describe pod -l app=api -n myapp | head -40

# The pod's logs
kubectl logs -l app=api -n myapp --tail=20

# The pod's exec (the command inside the container)
POD=$(kubectl get pods -n myapp -l app=api -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD -n myapp -- sh

# The pod's port's forwarding
kubectl port-forward -n myapp deployment/api 3000:3000 &
sleep 2
curl http://localhost:3000/
kill %1

# The pod's debug (the ephemeral container)
kubectl debug -it $POD -n myapp --image=busybox --target=api

# The service's connectivity's test
kubectl run test --rm -it --image=busybox -n myapp --restart=Never -- sh -c "
    wget -q -O - http://api:3000/ && echo 'The API is reachable'
    nslookup api.myapp.svc.cluster.local
    nslookup web.myapp.svc.cluster.local
"

# The DNS's resolution's test
kubectl run dns-test --rm -it --image=busybox -n myapp --restart=Never -- \
    nslookup db.myapp.svc.cluster.local

# The cluster's resource's usage
kubectl top nodes
kubectl top pods -A --sort-by=cpu | head -10
kubectl top pods -A --sort-by=memory | head -10

# The resource's requests and the limits
kubectl describe nodes | grep -A5 'Allocated resources'

# The capacity's and the allocatable's resources
kubectl get nodes -o json | \
    python3 -c "
import sys, json
data = json.load(sys.stdin)
for node in data['items']:
    print(f\"The node: {node['metadata']['name']}\")
    print(f\"  The capacity: {node['status']['capacity']}\")
    print(f\"  The allocatable: {node['status']['allocatable']}\")
"

# The persistent's volumes
kubectl get pv,pvc -A
kubectl get storageclasses

# The HPA's state
kubectl get hpa -A
kubectl describe hpa -n myapp

# The deployments' and the statefulsets' status
kubectl get deployments -A
kubectl get statefulsets -A
kubectl get daemonsets -A

# The services' and the ingresses' state
kubectl get services -A
kubectl get ingresses -A

# The secrets' state
kubectl get secrets -A

# The cluster's version and the upgrades
kubectl version --short
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kubeletVersion}{"\n"}{end}'

06 Interactive activity

07 Knowledge check

08 Unit discussion

09 Practice exercise

10 Summary

• The container orchestration is the automated's management of the containers' lifecycle, the placement, the networking, the storage, and the configuration across the clusters of the many's hosts. Its concerns: the scheduling, the scaling, the self-healing, the service's discovery, the rolling's updates, the configuration's management, the storage's management, and the observability.
• The Kubernetes's architecture: the control's plane (the API's server, the etcd, the scheduler, the controller's manager, the cloud's controller's manager) and the worker's nodes (the kubelet, the kube-proxy, the CRI's implementation, the CNI's plugins, the CSI's drivers).
• The Kubernetes's objects: the pods (the smallest's deployable's unit), the Deployments (the stateless's applications), the ReplicaSets (the pods' count), the Services (the stable's network's identities), the Ingresses (the external's HTTP and HTTPS), the ConfigMaps and the Secrets (the configuration), the StatefulSets (the stateful's applications), the DaemonSets (the every node's pods), the Jobs and the CronJobs (the batch's workloads), the namespaces (the logical's partitions), the RBAC (the authorization), and the network's policies (the traffic's rules).
• The Kubernetes's declarative's model: the operators declare the desired's state (the manifests), and the Kubernetes's controllers (the reconciliation's loops) make the actual's state match the desired's state. The kubectl's commands (the imperative's and the declarative's), the reconciliation's loops, and the GitOps's model (the manifests in the Git, the ArgoCD's and the Flux's tools).
• The Kubernetes's scheduling: the resource's requests and limits, the node's selection's rules (the nodeSelector, the nodeName, the affinity, the taints and the tolerations), the scheduler's policies (the filtering and the scoring), and the scheduler's profiles.
• The Kubernetes's networking: the network's model (the every pod has its own's IP), the CNI's plugins (the Calico's, the Cilium's, the Flannel's, the Weave's, the Antrea's, the Kube-router's), the services' networking (the ClusterIP's, the NodePort's, the LoadBalancer's, the kube-proxy's modes), and the DNS's resolution (the CoreDNS).
• The orchestration's alternatives: the Docker Swarm (the Docker's native's orchestration), the HashiCorp's Nomad (the HashiCorp's orchestration), the Apache Mesos (the Apache's cluster's manager), and the cloud's providers' services (the ECS's, the EKS's, the GKE's, the AKS's, the ACK's, the DOKS's).
• The orchestration's security: the cluster's layers (the cloud's infrastructure's, the cluster's, the node's, the pod's, the application's), the API's server's security (the authentication, the authorization, the admission, the audit's logging), the pod's security (the security's context, the Pod Security's Standards, the Pod Security's Admission), the network's security (the network's policies, the service's mesh's), the supply chain's security (the scanning, the signing, the attestation, the SBOM's, the admission's controllers), and the runtime's security (the Falco's, the Tetragon's, the Sysdig's).
• The orchestration's ecosystem: the Helm (the package's manager), the Kustomize (the configuration's management), the Istio (the service's mesh), the Prometheus (the monitoring), the ArgoCD and the Flux (the GitOps's tools), and the Operators (the applications' operational's knowledge's encapsulation).
• The orchestration's design's trade-offs: the declarative's model (the reproducibility, the versioning, the GitOps's, the self-healing's vs. the learning's curve, the debugging's difficulty), the scheduling's abstraction (the portability, the operations's automation vs. the control's loss), the pod's abstraction (the co-location vs. the complexity), the service's abstraction (the stable's identities vs. the networking's complexity), the cluster's abstraction (the scale, the multi-tenancy's vs. the management's complexity), and the ecosystem's diversity (the tools' choice vs. the integration's complexity).

11 References

• Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 18.
• Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, the Virtualization chapters. Available online at ostep.org.
• Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. Borg, Omega, and Kubernetes. ACM Queue, 2016.
• Bernstein, D. Containers and Cloud: From LXC to Docker to Kubernetes. IEEE Cloud Computing, 2014.
• Hightower, K., Burns, B., & Beda, J. Kubernetes: Up and Running, 3rd ed. O'Reilly, 2022.
• Lukša, M. Kubernetes in Action, 2nd ed. Manning, 2020.
• Kelsey Hightower's Kubernetes the Hard Way. The step-by-step's guide that builds the Kubernetes's cluster from the components.
• The Kubernetes's documentation (kubernetes.io/docs): the concepts, the tasks, the tutorials, the reference, and the best practices.
• The Cloud Native Computing Foundation (CNCF)'s landscape. The map of the orchestration's ecosystem.
• Rice, L. Container Security. O'Reilly, 2020.
• NIST SP 800-190. Application Container Security Guide. National Institute of Standards and Technology.
• Linux man-pages: man 1 kubectl, man 1 kubectl-apply, man 1 kubectl-get, man 1 kubectl-describe, man 1 kubectl-logs, man 1 kubectl-exec, man 1 kubectl-rollout, man 1 kubectl-scale, man 1 kubectl-port-forward, man 1 minikube, man 1 kind, man 1 k3s, man 1 helm, man 1 k9s, man 1 stern, man 1 kubectx, man 1 kubens.