COMP 3150Operating Systems

Module 1 · Unit 1 · Lesson 5 of 5

Operating-System Structures

The three major architectural families for organising kernel code — monolithic, microkernel, and hybrid — and the trade-offs that determine which design fits which system.

01 Introduction

The previous lesson examined the system-call interface: the boundary between user space and the kernel. This lesson looks at what is on the kernel side of that boundary. How is the kernel itself organised? What components does it contain, how are they connected, and what are the consequences of different design choices?

Operating-system architecture is a long-running debate. The central tension is between performance and isolation. A design that places all kernel functionality in a single address space is fast but exposes the whole system to a bug in any one part. A design that separates functionality into isolated components is safer but pays a performance penalty on every interaction.

Three families of design have emerged: monolithic kernels, microkernels, and hybrid kernels. Each is a response to this tension, and each has been used successfully in real systems — sometimes in the same product family. This lesson examines all three.

02 Learning objectives

By the end of this lesson you will be able to

  • Compare monolithic kernels.
  • Compare microkernels.
  • Compare hybrid kernels.
  • Analyse architectural trade-offs.
  • Evaluate operating-system designs.

This lesson contributes to U1.4, U1.5, and course learning outcome CLO1.

03 Required reading

  • Operating System Concepts — Chapter 2, Sections 2.7 and 2.8: Kernel Structures and Operating-System Structures.
  • Tanenbaum & Bos — Modern Operating Systems, Section 1.7: Operating System Structure.

04 Core content

4.1 Why architecture matters

An operating system is among the most complex pieces of software in existence. A modern Linux kernel contains on the order of 30 million lines of code; Windows and macOS are of comparable size. At that scale, the way code is organised — how components are grouped, how they communicate, and what privileges they hold — has consequences for correctness, performance, maintainability, and security.

Three architectural patterns dominate. Each answers the same question differently: where should kernel functionality live, and how should its parts talk to each other?

4.2 Monolithic kernels

In a monolithic kernel, all operating-system services run in a single address space in kernel mode. The scheduler, memory manager, file systems, network stack, device drivers, and system-call handlers are all part of one large program. Components call one another as ordinary function calls.

Canonical examples

Traditional UNIX, Linux, and the BSDs are monolithic. So, largely, are MS-DOS and early versions of macOS before the transition to a hybrid design.

Advantages.

  • Performance. Function calls within the kernel are cheap — a few cycles, not a mode switch. High-frequency operations benefit enormously.
  • Simplicity of interaction. Components can share data structures directly, without marshalling across a boundary.
  • Maturity. Decades of optimisation have gone into making monolithic kernels fast on real workloads.

Disadvantages.

  • Fragility. A bug in any part of the kernel can corrupt the entire system. A null-pointer dereference in a printer driver can bring down a server.
  • Security surface. Every kernel module is trusted absolutely. A compromised driver compromises the whole machine.
  • Complexity of maintenance. Interfaces between kernel subsystems can become entangled, and changes in one subsystem may require changes elsewhere.

The Linux exception

Modern Linux is monolithic but highly modular. Loadable kernel modules (LKMs) can be inserted and removed at runtime, and facilities like eBPF allow sandboxed code to run inside the kernel with strict verification. This blunts but does not eliminate the classic disadvantages.

4.3 Microkernels

A microkernel takes the opposite approach. Only the absolute minimum runs in kernel mode: typically low-level memory management, inter-process communication (IPC), and basic scheduling. Everything else — file systems, device drivers, network protocols, even the paging subsystem in the purest designs — runs as ordinary user-space server processes.

Canonical examples

Mach, QNX Neutrino, seL4, MINIX 3, and (in its original form) GNU Hurd. QNX is deployed widely in automotive and medical systems; seL4 is formally verified and used in high-assurance contexts.

Advantages.

  • Isolation. A crash in a driver kills only that server; the kernel can restart it. In QNX, a failed driver is typically restarted transparently.
  • Security. Because drivers and file systems are user-space processes, a compromised one cannot read arbitrary kernel memory. seL4's formal verification relies on this property.
  • Portability. Only the small microkernel needs to be ported to a new architecture.
  • Modularity. Servers can be added, replaced, or restarted without touching the kernel.

Disadvantages.

  • Performance. Every interaction between servers becomes an IPC message through the microkernel. What used to be a function call is now a series of context switches.
  • Complexity of design. Building a system out of many small communicating servers is intellectually cleaner but operationally harder than building one big program.
  • Historical performance gap. Early microkernels like first-generation Mach were notably slower than monolithic alternatives, and this reputation has been difficult to shake even as later designs (L4 and its descendants) largely closed the gap.

4.4 Hybrid kernels

A hybrid kernel combines elements of both designs. Most kernel functionality runs in kernel mode for performance — like a monolithic kernel — but the system is architected in a modular, message-passing style, and some subsystems can be moved into user space when isolation matters more than speed.

Canonical examples

Windows NT and its descendants (including Windows 10 and 11), macOS (XNU — combining Mach and BSD components), and to some extent modern Linux, which has absorbed ideas from both traditions.

The macOS XNU kernel is a particularly clear example. It embeds a Mach microkernel as the core and runs BSD file-system and networking code as kernel-mode servers, plus a driver framework (IOKit) and the ability to load additional kernel extensions. The design is hybrid in the sense that it takes the Mach IPC mechanism as its architectural foundation but does not push every service into user space.

Three kernel architectures compared Three panels. Left: monolithic kernel, all subsystems in one kernel-mode address space. Middle: microkernel, only IPC, scheduling, and basic memory in kernel mode, with drivers, file systems, and network as user-space servers. Right: hybrid kernel, with a small core plus kernel-mode subsystems that share message-passing architecture. MONOLITHIC user applications KERNEL MODE scheduler memory manager file systems network stack device drivers syscall handlers all in one address space direct function calls fast · fragile MICROKERNEL user applications USER-SPACE SERVERS drivers file sys network pager device manager MICROKERNEL IPC · scheduling basic memory interrupt handling minimal kernel surface message passing between servers robust · IPC overhead HYBRID user applications KERNEL MODE microkernel core IPC · scheduling · memory file systems (kernel-mode) network stack device drivers syscall handlers modular structure, mostly kernel-mode pragmatic balance used by Windows · macOS · Linux
Figure 1.5 — The three architectural families. The choice is not binary; modern systems combine elements of all three.

4.5 Additional architectural dimensions

The three families describe where code lives. Several other design decisions cut across all of them and are worth knowing.

Layered design
The system is organised into a hierarchy of layers, each built on the services of the layer below. The THE operating system (Dijkstra, 1968) is the classic example. Layering is clean in principle but inefficient in practice, because a request may traverse many layers to complete.
Modular design
The kernel is composed of loadable modules with well-defined interfaces. Linux's loadable kernel modules and Windows' driver framework are examples. This gives much of the flexibility of a microkernel without paying the IPC cost.
Exokernel design
A radical alternative in which the kernel multiplexes hardware securely but does almost nothing else; application-specific libraries implement the abstractions the application needs. Few production systems use this design, but the ideas influenced later research.
Virtual machine architectures
An OS runs on top of a hypervisor that presents a complete virtual hardware interface. The guest OS is unmodified, giving strong isolation. This is the foundation of cloud computing and is covered in Unit 13.

4.6 Evaluating the trade-offs

Choosing an architecture is not about picking "the best" one. It is about matching a design to a set of constraints. The following comparison is a starting point.

Architectural trade-offs at a glance
Property Monolithic Microkernel Hybrid
Kernel-mode code footprint Large Minimal Medium
Inter-component communication Direct function call IPC message Mix of both
Performance Highest Lowest (traditional) High
Fault isolation Weak Strong Moderate
Attack surface Large Small Medium
Portability of new drivers Rebuild kernel Restart server Load module
Complexity of design High in the large High in the small Varies
Representative systems Traditional UNIX, Linux QNX, seL4, MINIX 3 Windows, macOS

The trend line

Over time, the boundaries have blurred. Linux, historically monolithic, has adopted modular loading and sandboxed in-kernel programming (eBPF). Microkernels have been optimised (L4 family) to close much of the performance gap. Meanwhile, virtualization and containers have pushed some isolation concerns out of the kernel entirely and into the platform layer above it.

05 Worked example

Evaluate a specific architectural decision against a specific workload.

Should a network device driver live in kernel mode or user space?

  1. The workload. A 100 Gb/s network interface handling millions of small packets per second. Every packet must be delivered to the network stack and, ultimately, to an application.
  2. Monolithic position. Placing the driver in kernel mode eliminates a message-passing step per packet. For 10 million packets per second, saving even 200 nanoseconds per packet saves two seconds of CPU time per second of wall-clock time — an enormous efficiency gain.
  3. Microkernel position. Placing the driver in user space means a bug in the driver cannot corrupt the kernel. In a high-security context — say, an automotive ECU or a medical device — this may be the deciding factor even at a significant throughput cost.
  4. Hybrid position. Keep the fast path in kernel mode and place the management and configuration logic in user space. This is roughly what modern Linux does with the ethtool and driver interfaces.
  5. The decision. There is no universal answer. A datacentre switch and a pacemaker will legitimately make different choices for the same question.

How to evaluate any architectural decision

Ask three questions. What does the workload optimise for? What are the consequences of failure? What is the cost of the boundary being crossed? The answers determine whether the design's trade-offs are the right ones.

06 Interactive activity

Activity · inspect a real monolithic kernel

Linux is a monolithic kernel, but the modularity is visible from user space. Explore it in your VM.

  1. List loaded modules. Run lsmod | head -20. Each line is a kernel module loaded into the running kernel. Pick two and look them up with modinfo <name>.
  2. Check the kernel version and build info. Run uname -a and cat /proc/version.
  3. Load and unload a module. Run sudo modprobe dummy to load the dummy network driver. Verify with lsmod | grep dummy. Then run sudo modprobe -r dummy to unload it. This is the dynamic modularity that distinguishes modern Linux from classical monolithic designs.
  4. Inspect the kernel interface. List /proc and /sys. These are pseudo-filesystems through which user-space programs read and write kernel state. They blur the line between "kernel" and "user space" in a way that classical microkernel theory does not anticipate.
  5. Reflect. The Linux kernel is technically monolithic but behaves modularly. Which of the disadvantages of monolithic kernels from this lesson does this modularity mitigate? Which does it leave untouched?

07 Knowledge check

Check your understanding

Question 1

In a microkernel system, what mechanism replaces the direct function calls used inside a monolithic kernel, and why is this slower?

Show answer
It is replaced by inter-process communication (IPC). Because the file system, device drivers, and other services run as user-space processes, a request from one component to another must be delivered as a message routed through the microkernel. This involves at least two context switches (sender to kernel, kernel to receiver), a copy of the message contents, and additional bookkeeping. What costs a few dozen cycles as a function call can cost thousands as an IPC round trip.

Question 2

QNX is widely used in automotive safety systems. Which architectural property makes it attractive in that domain, and why?

Show answer
Microkernel fault isolation. In a car, a crashed driver must not take down the braking or steering systems. QNX runs device drivers as user-space processes, so if one fails, the microkernel can restart it while the rest of the system continues operating. The performance cost of IPC is acceptable in exchange for predictable, contained failure.

Question 3

Windows and macOS both run kernel-mode device drivers, yet both are described as hybrid rather than monolithic. What features of their design justify the hybrid label?

Show answer
Both systems use a microkernel-style core — Mach in the case of macOS, the NT executive in the case of Windows — around which other subsystems are layered with well-defined interfaces. macOS in particular routes many operations through the Mach IPC mechanism even when the server runs in kernel mode. The result is a design that keeps monolithic-style performance for the fast path while retaining a modular, message-passing architecture at the structural level. That combination — monolithic in placement, microkernel-inspired in structure — is the defining characteristic of a hybrid.

08 Unit discussion

Discussion prompt

Linux is monolithic and has been extraordinarily successful. Microkernels are architecturally cleaner and, in principle, more secure. Yet Linux is deployed far more widely.

What best explains this outcome? Consider performance, ecosystem effects, developer familiarity, and the historical moment at which each design was introduced. Argue for whichever factor you consider most important, and acknowledge the counter-argument.

Post your response to the Unit 1 discussion forum and reply to at least two classmates. This activity contributes to your participation grade and supports CLO1 and CLO7.

09 Practice exercise

  1. Design a kernel. You are designing an operating system for an implantable medical device with strict reliability requirements and modest performance needs. Would you choose monolithic, microkernel, or hybrid? Write a 300-word rationale.
  2. Design a kernel for the opposite case. Now design for a database server handling 500 000 queries per second. Would your answer change? Why or why not?
  3. Read about a real microkernel. Look up the seL4 microkernel. What does "formally verified" mean in this context, and what does the verification prove? Which class of bugs does it eliminate?
  4. Compare driver models. Compare how Linux and macOS handle device drivers. Identify at least three structural differences and explain what each implies for performance and stability.

10 Summary

  • Three architectural families dominate: monolithic (all services in kernel mode, direct function calls), microkernel (minimal kernel, most services in user space, IPC between them), and hybrid (a microkernel-style core plus kernel-mode subsystems).
  • Monolithic kernels prioritise performance; microkernels prioritise isolation and security; hybrid kernels balance the two.
  • The choice is not binary, and modern systems combine elements of all three. Linux is monolithic but modular; macOS is hybrid with a Mach core; QNX is a pure microkernel deployed in safety-critical contexts.
  • Additional architectural dimensions — layered, modular, exokernel, virtual-machine architectures — cut across these families.
  • Evaluating any architectural decision means asking what the workload optimises for, what the consequences of failure are, and what the cost of crossing the boundary is.

Unit 1 complete

This closes the theoretical foundation of Module 1. Unit 2 turns to the practical environment — Linux and UNIX — in which the rest of the course will be carried out.

11 References

  • Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, Chapter 2.
  • Tanenbaum, A. S., & Bos, H. Modern Operating Systems, Chapter 1.
  • Ritchie, D. M., & Thompson, K. The UNIX Time-Sharing System. Communications of the ACM, 1974.
  • Liedtke, J. On µ-Kernel Construction. SOSP, 1995.
  • Klein, G., et al. seL4: Formal Verification of an OS Kernel. SOSP, 2009.
👤