Lesson 11.1: I/O Hardware

01 Introduction

Unit 10 presented the file system as a set of logical structures: the file and the directory (Lessons 10.1–10.4), the allocation of a file's blocks (Lesson 10.5), the journal that keeps the metadata consistent (Lesson 10.6), and the modern file systems that combine the concepts (Lesson 10.7). The unit treated the storage device as an abstraction — a sequence of numbered blocks that the file system could read and write. This unit removes the abstraction. It presents the I/O subsystem: the hardware devices, the controllers that drive them, the buses that connect them to the processor, and the kernel layers — the interrupt handlers, the DMA machinery, the device drivers, and the scheduling policies — that turn the file system's logical requests into the device's physical operations.

The I/O subsystem is the operating system's largest and most heterogeneous component. A modern computer has dozens of I/O devices: storage devices (hard disks, solid-state drives, optical drives), input devices (keyboards, mice, touchscreens, scanners), output devices (displays, printers, speakers), network devices (Ethernet controllers, Wi-Fi adapters), and the devices that support them (USB controllers, PCI Express bridges, timers, real-time clocks). The devices differ enormously in their characteristics: some transfer a few bytes per second, others transfer tens of gigabytes per second; some require the processor's constant attention, others run autonomously; some are addressed by port number, others by memory address. The I/O subsystem's challenge is to provide a uniform interface to this heterogeneous collection, so that applications and the file system can use the devices without knowing their details.

The uniformity is achieved by layering. At the top, the application issues a system call — read, write, ioctl — and the kernel's file system or device-independent layer translates the call into a device-specific request. In the middle, the device driver translates the request into the sequence of register operations and memory transfers that the device requires. At the bottom, the device controller — the hardware that drives the device — executes the operations and reports their completion. The layers are connected by the bus, and the data moves between the device and memory by one of three mechanisms: programmed I/O (the processor transfers each byte or word itself), interrupt-driven I/O (the processor issues a command and the device interrupts it when the command completes), and direct memory access (a separate controller transfers the data between the device and memory without the processor's involvement).

This lesson presents the I/O subsystem's hardware. It defines the I/O subsystem and its layers. It presents the device and the device controller, the registers through which the processor commands the device, and the two addressing schemes — port-mapped and memory-mapped I/O. It presents the three data-transfer mechanisms and explains when each is appropriate. It presents the bus hierarchy (PCI Express and the storage-specific buses) and the device classes (block, character, and network) that the kernel uses to organize the devices. It presents device addressing and enumeration, and it presents the hard disk's geometry — the device that the storage stack was originally designed for — and the solid-state drive's characteristics that the modern stack must accommodate. It closes with the hardware characteristics that the I/O software must hide.

Prerequisites

This lesson builds on Unit 10 and on the earlier units. The file system's block-level view of storage (Lesson 10.5) and the disk-related terminology (sectors, blocks, extents) are assumed. The interrupt mechanism from the process-management unit (Unit 3, Lesson 3.5) is the basis for the discussion of interrupt-driven I/O, and the memory hierarchy from the memory-management unit (Lesson 8.1) is the basis for the discussion of DMA and the I/O bus. No new operating-system concepts are introduced; the lesson is about the hardware and the lowest kernel layers. Familiarity with the Linux command line from Lesson 2.2 is helpful, since the interactive activity uses standard Unix tools.

02 Learning objectives

By the end of this lesson you will be able to

Describe the I/O subsystem and the layers through which an application's I/O request passes.
Explain the role of the device controller and the device registers, and compare port-mapped and memory-mapped I/O.
Describe programmed I/O, interrupt-driven I/O, and direct memory access, and explain when each is appropriate.
Describe the bus hierarchy, including PCI Express and the storage buses, and explain how devices are enumerated and addressed.
Compare the block, character, and network device classes and explain how the kernel uses the classes to organize the device drivers.
Describe a hard disk's geometry and the characteristics of a solid-state drive, and explain how the differences affect the I/O software.

This lesson contributes to U11.1 and U11.2, and to course learning outcomes CLO1, CLO5, and CLO6.

03 Required reading

Operating System Concepts — Chapter 12, I/O Systems, Sections 12.1–12.3 and 12.7. The standard treatment of the I/O hardware, the device controller, the data-transfer mechanisms, and the bus hierarchy.
OSTEP — Chapter 36, I/O Devices. OSTEP's treatment of the device interface, the canonical device protocol, and the three data-transfer mechanisms (programmed I/O, interrupts, and DMA) is the clearest available account, and it is the basis for the worked example in this lesson.
Tanenbaum & Bos — Modern Operating Systems, Chapter 5, Input/Output, Sections 5.1–5.3. A complementary treatment that develops the I/O hardware from first principles and compares the data-transfer mechanisms systematically.
Love, R. — Linux Kernel Development, 3rd ed., Chapter 14, Block Devices I/O, and Chapter 16, The Block I/O Layer. The Linux kernel's block-device structures, the request queue, and the I/O scheduler's interface.
Bovet, D. P., & Cesati, M. — Understanding the Linux Kernel, 3rd ed. O'Reilly, 2005. Chapter 13, I/O Architecture and Device Drivers. The Linux kernel's I/O architecture, the bus structures, and the device model.
Kerrisk, M. — The Linux Programming Interface, Chapter 4, File I/O: The Universal I/O Model, and Chapter 13, File I/O Buffering. The application's view of the I/O interface and the kernel's buffering.
Patterson, D. A., & Hennessy, J. L. — Computer Organization and Design. The treatment of the I/O bus, the interrupt mechanism, and the memory-mapped I/O from the computer architecture perspective.
Linux man pages: man 4 tty, man 4 sd, man 4 null, man 4 zero, man 8 lspci, man 8 lsusb, man 1 lsblk, man 1 lscpu, man 5 proc (for /proc/interrupts, /proc/ioports, and /proc/iomem), man 8 hdparm, man 8 smartctl, man 1 iostat.

Reading strategy

Read the OSTEP chapter first. It develops the I/O device interface from a concrete example — a canonical device with a status register, a command register, a data register, and an interrupt — and shows step by step how the operating system uses the registers, first with programmed I/O, then with interrupts, then with DMA. The progression is the chapter's central argument, and it is the structure this lesson follows. Then read the Silberschatz section for the formal classification of the I/O hardware: the device controller, the port-mapped and memory-mapped addressing, the polling and interrupt mechanisms, and the DMA. The Tanenbaum chapter is the complementary view, and it is worth reading for the discussion of the I/O software layers. Finally, the Linux kernel chapters (Love and Bovet & Cesati) are the practical view: they show how the kernel's device model, bus structures, and block layer implement the abstractions that the other readings describe. Pay particular attention to the difference between the device (the mechanical or electronic component) and the controller (the hardware that drives the device and presents it to the processor). The distinction is the key to understanding the layering: the processor never talks to the device directly; it always talks to the controller.

04 Core content

4.1 The I/O subsystem and its layers

The I/O subsystem is the collection of hardware and software that connects the processor and memory to the I/O devices. Its purpose is to move data between the devices and memory, and to provide a uniform interface that the rest of the operating system can use without knowing the devices' details.

The subsystem is organized as a stack of layers. An application's request passes down through the layers, and the device's response passes back up:

Application: The program that issues the I/O request. It uses the operating system's I/O interface — the system calls read, write, open, close, ioctl — or a library that wraps them (the C standard library's fread and fwrite, a language runtime's BufferedReader, and so on).
Device-independent I/O software: The kernel layer that provides the common interface: the file system's VFS (the virtual file system), the buffering and caching, the error handling, and the device naming. It translates the application's request — "read 4 KB from offset 1 MB of this file" — into a request on a specific device — "read blocks 2,048 through 2,055 of this partition".
Device driver: The kernel module that knows how to operate a specific device or class of devices. It translates the device-independent request into the sequence of register operations and memory transfers that the device's controller requires, and it handles the device's interrupts and errors.
Device controller: The hardware that drives the device. It receives commands from the processor (through its registers), executes them (by operating the device's mechanical or electronic components), and reports their completion (through its registers and its interrupts).
Device: The mechanical or electronic component that performs the physical operation: the disk's platters and heads, the SSD's flash chips, the keyboard's keys, the display's pixels. The device is connected to the controller, and the controller presents it to the processor as a set of registers.

The layering is the subsystem's central design principle. The application does not know which device holds its data; the file system does not know how the device's controller works; the driver does not know why the data is being read. Each layer translates between the layer above and the layer below, and each layer can be replaced — a new device requires a new driver, not a new file system; a new file system requires a new VFS implementation, not a new device driver — without disturbing the others.

4.2 Devices and device controllers

A device is a piece of hardware that performs an I/O operation. It is connected to the computer by a controller (also called an adapter), a piece of electronics that operates the device on the processor's behalf. The processor never talks to the device directly; it talks to the controller, and the controller talks to the device.

The distinction matters because the controller is the device's interface. A disk controller, for example, accepts commands such as "read sector 1,234,567" or "write sector 1,234,567 with this data", and it operates the disk's heads, platters, and motors to carry them out. The processor issues the commands by writing to the controller's registers, and it learns the results by reading the same registers. The device's physical details — the head's position, the platter's rotation, the flash chip's erase-block size — are hidden behind the controller's register interface.

A controller typically has four kinds of registers:

Data registers. The registers through which the data is transferred. On a programmed-I/O device, the processor writes each byte or word to the data register and the controller passes it to the device (or reads each byte or word from the register and returns it to the processor). On a DMA-capable device, the data register is bypassed: the controller reads and writes memory directly, and the data register is used only for small control transfers.
Control registers. The registers through which the processor issues commands. A write to the control register tells the device what to do: read, write, seek, reset, and so on. The commands are encoded as bit patterns or as small integers.
Status registers. The registers through which the device reports its state. A read of the status register returns a bit pattern that indicates whether the device is ready, busy, or in an error state, and often includes additional information such as the number of bytes transferred or the error's cause.
Interrupt-related registers. The registers that control the device's interrupt behaviour: whether interrupts are enabled, which interrupt line the device uses, and how the interrupt is acknowledged. These registers are discussed in detail in Lesson 11.2.

The registers are the processor's window into the device. Every operation the processor performs on the device is a read or a write of one of these registers, and the device's behaviour is entirely determined by the sequence of register operations the driver performs. A driver that writes the wrong value to the control register can command the device to do the wrong thing; a driver that reads the status register at the wrong time can miss the device's completion or its error.

4.3 Port-mapped and memory-mapped I/O

The processor must be able to address the device's registers, and there are two ways it can do so: port-mapped I/O and memory-mapped I/O.

Port-mapped I/O (also called isolated I/O) gives the device's registers their own address space, separate from memory. The processor has special instructions — on x86, the IN and OUT instructions — to read and write the port space, and the bus carries a signal that distinguishes a port access from a memory access. The port space is small (65,536 ports on x86), and the device's registers occupy a few ports within it.

Memory-mapped I/O maps the device's registers into the processor's memory address space. A range of memory addresses is reserved for the device's registers, and the processor uses its ordinary load and store instructions to read and write them. The memory controller or the bus's address decoder recognizes the addresses and routes the accesses to the device instead of to memory.

The two schemes have different trade-offs:

Memory-mapped I/O is simpler for the software. The driver can use the processor's ordinary load and store instructions, and the compiler can generate them without special intrinsics or inline assembly. The driver can also use the processor's addressing modes (indexed, indirect, and so on) to access the registers, which is convenient for devices with arrays of registers.
Memory-mapped I/O is faster on modern hardware. The processor's load and store instructions are optimized for the memory hierarchy, and they can be pipelined and reordered more aggressively than the special I/O instructions. The special instructions of port-mapped I/O are serializing on many processors, so they are slower.
Memory-mapped I/O consumes address space. The device's registers occupy memory addresses that could otherwise be used for memory. On a 64-bit processor the address space is large enough that the loss is irrelevant; on a 32-bit processor the loss is sometimes significant.
Port-mapped I/O is isolated. The device's registers cannot be confused with memory, and a wild pointer cannot accidentally write to a device's control register. The isolation is a safety property, but it is not a strong one: a driver can still write to a device's ports deliberately, and the port space is not protected from user-mode access on all processors.

Modern systems use memory-mapped I/O almost exclusively. PCI Express devices are memory-mapped by design, and the port-mapped space is reserved for legacy devices and for the processor's own control registers. The Linux kernel's device drivers use ioremap to map a device's registers into the kernel's address space, and they use readl, writel, and their relatives to access them.

Memory-mapped I/O is not ordinary memory

A memory-mapped device register looks like a memory location, but it does not behave like one. A read of a register may have a side effect (it may clear a status bit or advance a queue); a write may have a side effect (it may start an operation); and two reads of the same register may return different values even if nothing else has written to it. The compiler and the processor do not know this, so they may reorder, combine, or eliminate accesses to the registers as if they were ordinary memory accesses. The driver must prevent this: it uses volatile accesses (or the kernel's accessor functions, which include the necessary barriers) to ensure that each access happens exactly once and in the intended order. A driver that treats a device register as ordinary memory is a driver with a subtle and intermittent bug.

4.4 Programmed I/O and polling

The simplest way to transfer data between the processor and a device is programmed I/O: the processor transfers each byte or word itself, by reading from or writing to the device's data register. The processor is fully occupied during the transfer, and its throughput is limited by the rate at which it can execute the load and store instructions.

Programmed I/O is usually combined with polling: after the processor issues a command to the device, it repeatedly reads the device's status register to check whether the operation has completed. When the status register indicates that the device is ready, the processor transfers the next byte or word. The loop continues until the entire transfer is complete.

Polling's advantage is its simplicity: it requires no interrupts, no interrupt handler, and no interrupt controller, and its behaviour is entirely predictable. Its disadvantage is its waste: the processor spends the entire transfer in the polling loop, doing nothing but reading the status register, and the time it spends is proportional to the device's latency rather than to the amount of data transferred. For a slow device — a keyboard, a serial port at 9,600 bits per second — the waste can be enormous: the processor may execute thousands of instructions waiting for a single character.

Polling is nevertheless the right choice in some situations:

When the device is fast. A solid-state drive that responds in a few microseconds is faster than the interrupt overhead, so polling may be cheaper than taking an interrupt. Modern high-performance drivers (NVMe, high-speed networking) use polling for exactly this reason.
When the processor has nothing else to do. In an embedded system with a single task, polling is simpler than an interrupt-driven design and just as effective.
When the transfer is short. For a few bytes, the interrupt's overhead (saving the processor's state, entering the handler, returning) may exceed the transfer's cost. The driver may poll for the first few microseconds and fall back to interrupts if the device does not respond promptly.

The general rule is that polling is appropriate when the expected wait is short and the processor has no other work to do, and that interrupts (or DMA) are appropriate when the expected wait is long or the processor is busy.

4.5 Interrupt-driven I/O

Interrupt-driven I/O solves polling's waste by letting the processor do other work while the device operates. The processor issues a command to the device and continues with its other tasks. When the device completes the command, its controller raises an interrupt, which causes the processor to suspend its current task, save its state, and jump to the device's interrupt handler. The handler reads the device's status register, transfers any remaining data, and returns the processor to the interrupted task.

The mechanism's details are the subject of Lesson 11.2. For this lesson, the important points are:

The interrupt is a hardware signal. The device's controller asserts a signal on an interrupt line, and the processor's interrupt controller (the APIC on modern x86 systems, the GIC on ARM systems) receives it and delivers it to the processor.
The interrupt is asynchronous. It can arrive at any time, in the middle of any instruction, and the processor's response is to suspend the current instruction stream and enter the handler. The handler must therefore be prepared for any state of the system.
The interrupt is prioritized. A processor can be interrupted by a higher-priority interrupt while it is handling a lower-priority one, and it can mask (disable) interrupts of a given priority while it is performing a critical operation. The prioritization and masking are what make the interrupt mechanism usable in a system with many devices and many concurrent operations.
The interrupt has overhead. Saving the processor's state, entering the handler, identifying the device (the interrupt controller may report the source, or the handler may have to poll the devices), and returning to the interrupted task take time — typically a few hundred to a few thousand cycles. For a device that interrupts frequently, the overhead can consume a significant fraction of the processor's time.

Interrupt-driven I/O is the standard mechanism for slow and medium-speed devices: keyboards, mice, serial ports, network adapters at moderate rates, and storage devices whose transfers are small. It is less appropriate for devices that transfer large amounts of data at high rates, because each interrupt transfers only a small amount of data (the processor reads or writes the data register a few times per interrupt) and the interrupt overhead dominates. For those devices, DMA is the answer.

4.6 Direct memory access

Direct memory access (DMA) transfers data between a device and memory without the processor's per-byte involvement. The processor sets up the transfer by writing the memory address, the byte count, and the direction to a DMA controller (which may be a separate chip or, on modern systems, a part of the device's own controller). The DMA controller then performs the transfer, reading from or writing to memory directly, and raises an interrupt when the transfer is complete.

The DMA mechanism has three parts:

Setup. The processor's driver prepares the transfer: it allocates a memory buffer, ensures that the buffer is physically contiguous (or uses a scatter-gather list, described below), translates the buffer's virtual address to a physical address, and writes the address, the count, and the direction to the DMA controller's registers. The setup is a small number of register writes, and the processor is occupied only for the setup's duration.
Transfer. The DMA controller transfers the data. It arbitrates with the processor for the memory bus (or the memory controller's bandwidth), reading or writing one word at a time. The processor continues to execute instructions, although its memory accesses may be delayed by the DMA controller's use of the bus. On modern systems the arbitration is handled by the memory controller and the bus's protocol, and the processor's slowdown is usually small.
Completion. When the transfer is complete, the DMA controller raises an interrupt. The driver's handler checks the transfer's status, performs any cleanup, and signals the waiting process (or completes the asynchronous I/O operation).

DMA's advantage is the processor's freedom. During the transfer, the processor can execute other instructions, so the device's latency and the transfer's duration are overlapped with useful work. For a device that transfers megabytes at a time — a disk, an SSD, a gigabit network adapter — the saving is enormous: the processor spends a few hundred cycles setting up the transfer and a few hundred handling the completion, instead of millions of cycles moving the data.

The complication is the physical address problem. A DMA controller accesses memory by physical address, but a program's buffer is specified by virtual address, and the virtual address's pages may be scattered across physical memory. The operating system handles the problem in one of three ways:

Contiguous allocation. The kernel allocates a physically contiguous buffer for the transfer, copies the data between the buffer and the program's scattered pages, and performs the DMA on the contiguous buffer. The approach works for any device but costs an extra copy, which is significant for high-bandwidth transfers.
Scatter-gather DMA. The device's controller accepts a scatter-gather list, a list of (physical address, length) pairs that describe the buffer's pages. The controller reads the list and transfers each page separately, so no copy is needed. The approach is the standard for modern storage and network controllers, and it is the reason the kernel's block layer can submit a request that spans many non-contiguous pages.
IOMMU. The I/O memory management unit is a hardware component that translates the device's addresses the way the processor's MMU translates the program's addresses. The kernel programs the IOMMU with a mapping from the device's address space to physical memory, and the device uses virtual addresses (from its own perspective) that the IOMMU translates. The IOMMU also provides isolation: a device can access only the memory that the kernel has mapped for it, which prevents a buggy or malicious device from corrupting arbitrary memory.

The choice among the three approaches depends on the hardware and the transfer's size. Modern systems use scatter-gather DMA with an IOMMU when the hardware supports it, contiguous allocation as a fallback, and programmed I/O or interrupts for small transfers where the DMA setup's cost would exceed the transfer's benefit.

Figure 11.1 — The I/O subsystem's layers and the three data-transfer mechanisms. The left column of the upper diagram is the layer stack: an application's request passes down through the device-independent software and the device driver to the device controller, which operates the device. The right column is the bus, which carries the addresses, data, interrupts, and control signals between the controller and the processor and memory. The lower diagram compares the three transfer mechanisms: programmed I/O and polling keep the processor busy for the whole transfer; interrupt-driven I/O frees the processor during the wait but pays an interrupt per transfer; DMA frees the processor for the whole transfer but requires a setup and a physical-address translation.

4.7 The bus hierarchy

The bus is the set of wires (or, on modern systems, the high-speed serial links) that carry data, addresses, and control signals between the processor, memory, and the devices' controllers. The bus is the I/O subsystem's physical backbone: the device controller is connected to the bus, and the processor communicates with the controller by addressing it on the bus.

Modern computers use a hierarchy of buses rather than a single bus, because a single bus cannot satisfy all the devices' requirements. A typical hierarchy has three levels:

The memory bus. The bus between the processor and main memory. It is the fastest bus in the system — tens of gigabytes per second on modern systems — and it is optimized for the processor's memory accesses. Devices are not connected to the memory bus directly; they reach memory through a bridge.
The general-purpose I/O bus. The bus that connects the high-bandwidth devices — graphics adapters, storage controllers, network adapters — to the processor and memory. On modern systems this is PCI Express (PCIe), a high-speed serial bus that replaced the parallel PCI bus in the mid-2000s. PCIe is organized as a set of lanes, and a device can use one lane (×1), four lanes (×4), sixteen lanes (×16), or more, depending on its bandwidth requirement.
The low-speed expansion buses. The buses that connect the slow devices — keyboards, mice, printers, external storage — to the system. USB is the standard for external devices; SATA and SAS are the buses for internal storage devices; NVMe is the protocol that runs over PCIe for high-performance SSDs.

The hierarchy's purpose is to match each device to a bus that suits its bandwidth and cost. A keyboard does not need a PCIe ×16 connection, and a graphics adapter cannot use USB. The bridges between the buses translate the protocols, so a device on a low-speed bus can still be addressed by the processor and can still interrupt it. The translation's cost is the reason the hierarchy is arranged with the fastest devices closest to the processor: a device on a slower bus pays more latency per access.

The storage buses deserve particular attention because they are the subject of the rest of this unit:

SATA (Serial ATA) is the standard bus for consumer hard disks and SSDs. It transfers data at up to 6 Gb/s (about 600 MB/s after encoding), and it supports a single device per port. Its command protocol (AHCI) is the legacy protocol that the operating system uses to issue read and write commands.
SAS (Serial Attached SCSI) is the enterprise equivalent of SATA. It transfers data at up to 22.5 Gb/s, supports multiple devices per port (through expanders), and provides the SCSI command set, which is richer than AHCI's. It is used in servers and storage arrays.
NVMe (Non-Volatile Memory Express) is the protocol that runs over PCIe for high-performance SSDs. It transfers data at the PCIe bus's rate (tens of gigabytes per second), and it provides a queue-based interface: the device has thousands of command queues, each of which can hold thousands of commands, and the operating system submits commands by writing them to a queue in memory. The queue-based interface is designed for the parallelism of flash storage, and it is the reason NVMe SSDs achieve far higher throughput and lower latency than SATA SSDs.

The bus's bandwidth and latency determine the I/O subsystem's performance ceiling. A storage device connected to a SATA port cannot exceed SATA's bandwidth, no matter how fast the device is; an NVMe SSD can use the PCIe bus's full bandwidth. The bus's characteristics are therefore an important part of the I/O subsystem's design, and the operating system's drivers must be aware of them: a driver that submits commands faster than the bus can carry them wastes the processor's time, and a driver that submits them too slowly leaves the bus idle.

4.8 Device classes

The kernel organizes the I/O devices into a small number of device classes, each with its own interface and its own driver model. The classical Unix division has three classes:

Block devices: Devices that transfer data in fixed-size blocks and support random access: hard disks, solid-state drives, optical drives, and the logical volumes and RAID arrays built on them. A block device's interface is read and write at a block offset, and the kernel caches the device's blocks in the page cache. The block device is the file system's substrate: a file system is mounted on a block device, and its blocks are the device's blocks.
Character devices: Devices that transfer data as a stream of bytes and do not support random access: keyboards, mice, serial ports, terminals, and the pseudo-devices /dev/null, /dev/zero, and /dev/random. A character device's interface is read and write at the stream's current position, and the kernel does not cache the data. A character device's operations are often unbuffered in the sense that each read or write is passed directly to the device driver.
Network devices: Devices that transfer data as packets and are addressed by network address rather than by block or byte offset: Ethernet controllers, Wi-Fi adapters, and the virtual interfaces that the kernel creates (loopback, bridges, tunnels). A network device's interface is the socket API rather than read and write, and the kernel's networking stack — not the file system — is the layer above the driver.

The classes are not rigid. Some devices have both a block and a character interface: a SCSI tape drive, for example, is accessed as a block device for its sequential data and as a character device for its control operations. Some devices belong to none of the three classes: a display adapter, a sound card, and a hardware random-number generator each have their own interface. The kernel's device model (the Linux struct device and the sysfs filesystem) accommodates the variety by describing the devices' properties — their parent bus, their driver, their power state — in a uniform way, while the drivers provide the class-specific operations.

The classes matter because they determine the kernel's caching and buffering strategy. A block device's accesses go through the page cache, so repeated reads of the same block are served from memory; a character device's accesses bypass the page cache, so each read reaches the device; a network device's packets go through the socket buffers, so the kernel's networking stack can reassemble, reorder, and route them. The choice of class is therefore a statement about the device's semantics, and the kernel's behaviour follows from it.

4.9 Device addressing and enumeration

For the processor to communicate with a device, it must be able to address it. The addressing has several levels:

The bus address. The device's position on its bus: a PCIe device's bus number, device number, and function number; a SCSI device's target and logical unit number (LUN); a USB device's port and endpoint. The bus address is what the device's controller uses to recognize the processor's accesses.
The register address. The address (in the port space or the memory space) of the device's registers. On a PCIe device, the registers are described by a BAR (base address register), which the system's firmware or the kernel assigns during enumeration.
The interrupt number. The interrupt line or message that the device uses to signal its completion. On modern systems, the interrupt is a message-signaled interrupt (MSI or MSI-X): the device writes a specific value to a specific memory address, and the interrupt controller interprets the write as an interrupt.
The device node. The name by which the kernel presents the device to user space: the /dev/sda block device, the /dev/ttyS0 character device, the eth0 network interface. The device node is created by the kernel (on modern systems, by udev in response to the kernel's events) and is the handle that applications use.

The process of discovering the devices and assigning their addresses is called enumeration. On a modern system, enumeration is a collaboration between the firmware (the UEFI or BIOS), the buses' controllers, and the kernel:

The firmware enumerates the buses. The UEFI firmware scans the PCIe buses, discovers the devices, assigns each device's bus address and BAR, and builds the ACPI tables that describe the system's devices. The firmware's work is passed to the kernel in the ACPI tables and the device tree.
The kernel enumerates the buses again. The Linux kernel re-scans the buses during its initialization, discovers the devices that the firmware did not report (or that were hot-plugged after the firmware's scan), and assigns their addresses. The kernel's enumeration is authoritative on modern systems; the firmware's enumeration is a starting point.
The kernel matches devices to drivers. Each device has an identifier — a PCI vendor and device ID, a USB class code, an ACPI hardware ID — and the kernel's driver model matches the identifier to a driver's table of supported devices. When a match is found, the driver's probe function is called; the function initializes the device and registers it with the kernel's device model.
The kernel creates the device nodes. The kernel's device model creates the entries in sysfs and (through udev) the nodes in /dev. The device is now available to user space.

Enumeration's importance is that it makes the system self-configuring. A modern computer can accept a new device — a USB drive, a PCIe card, a SATA disk — and discover, configure, and drive it without the administrator's intervention. The self-configuration is the result of the buses' standardized discovery protocols and the kernel's driver model, and it is one of the operating system's most visible accomplishments: the user plugs in the device, and it works.

4.10 Hard-disk geometry

The hard disk is the device that the storage stack was originally designed for, and its characteristics shaped the file system's and the I/O subsystem's design.

A hard disk's physical components are:

Platters. The rotating disks coated with magnetic material. A typical disk has one to five platters, each with two surfaces (one per side).
Heads. The read/write heads, one per platter surface, mounted on a moving arm. The heads fly a few nanometres above the platter's surface and read or write the magnetic patterns.
Tracks. The concentric circles on a platter's surface. A track holds tens to hundreds of kilobytes of data.
Sectors. The subdivisions of a track. The classical sector size is 512 bytes; the modern sector size is 4,096 bytes (the Advanced Format), and modern drives often report 512-byte sectors while using 4,096-byte sectors internally (the 512e scheme).
Cylinders. The set of tracks at the same radial position on all the platters' surfaces. The cylinder is the unit that the classical disk scheduling algorithms (Lesson 11.5) reason about.

The disk's geometry determines its performance characteristics. Reading or writing a sector requires three operations:

Seek. Moving the heads to the correct cylinder. The seek time is a few milliseconds on a modern disk, and it is the largest component of the access time for random accesses.
Rotational delay. Waiting for the platter's rotation to bring the correct sector under the head. The delay averages half a rotation; at 7,200 revolutions per minute (120 rotations per second), half a rotation is about 4.2 milliseconds.
Transfer. Reading or writing the sector's data. The transfer time is a few microseconds per sector at the disk's sequential bandwidth (a few hundred megabytes per second).

The three components' relative sizes explain the disk's performance profile. A random read costs a seek plus a rotational delay plus a transfer — a few milliseconds, of which the transfer is a tiny fraction. A sequential read costs one seek and one rotational delay for the first sector, and then only the transfer time for the rest, because the heads do not move and the platter brings the sectors under them in order. The disk's sequential bandwidth can be hundreds of megabytes per second, while its random throughput is limited to a few hundred operations per second (the reciprocal of the access time). The difference is the reason the file system's allocation policies (Lesson 10.5) try to place a file's blocks contiguously, and the reason the disk scheduling algorithms (Lesson 11.5) try to reduce the number of seeks.

4.11 Solid-state drives

A solid-state drive (SSD) stores data in flash memory rather than on magnetic platters. Its characteristics differ from a hard disk's in ways that matter to the I/O subsystem:

No mechanical delay. An SSD has no heads to seek and no platter to rotate, so its access time is dominated by the flash memory's read latency (tens of microseconds) and the controller's overhead. The difference between sequential and random access is far smaller than on a hard disk.
Erase-before-write. Flash memory cannot be overwritten in place: a block must be erased before it can be written, and the erase operation works on a larger unit (the erase block, typically 256 KB to several megabytes) than the read/write unit (the page, typically 4 KB to 16 KB). The SSD's controller hides the constraint behind a flash translation layer (FTL), which maps the logical block addresses to physical flash pages and performs the erase operations when necessary.
Wear. Each flash cell can be erased and rewritten only a limited number of times (typically a few thousand cycles for consumer drives, more for enterprise drives). The FTL performs wear leveling, spreading the writes evenly across the cells, so the drive's lifetime is determined by the total number of writes rather than by the hot spots' exhaustion. The operating system can help by telling the drive which blocks are free (the TRIM command), so the FTL does not have to copy them during garbage collection.
Write amplification. The FTL's operations — reading a page, modifying it, and writing it back to a new location — cause more physical writes than the logical writes the operating system requested. The ratio is the write amplification factor, and it determines the drive's performance and lifetime. The operating system's I/O patterns affect the factor: sequential writes and aligned writes cause less amplification than random writes and unaligned writes.
Parallelism. An SSD contains many flash chips, and the controller can access several of them in parallel. The parallelism is the source of the SSD's high throughput, and it is the reason the drive can satisfy many concurrent requests without degrading. The NVMe protocol's queue-based interface is designed to exploit the parallelism: the operating system can submit thousands of commands at once, and the drive's controller can schedule them across its chips.

The SSD's characteristics have reshaped the I/O subsystem's design. The disk scheduling algorithms (Lesson 11.5) that minimize seeks are less important on SSDs, where the access time is nearly uniform, but the algorithms that maximize parallelism and minimize write amplification are more important. The file systems' allocation policies (Lesson 10.5) that try to place data contiguously are still useful (they reduce the FTL's work), but the policies that try to reduce seeks are less so. The flash-specific file systems (F2FS, Lesson 10.7) are designed around the SSD's constraints, and the general-purpose file systems' flash-friendly modes (the discard option, the periodic fstrim) are designed to cooperate with the FTL rather than to fight it.

The storage stack's two worlds

The I/O subsystem must accommodate both worlds. A hard disk's performance depends on the order of the requests: a sequence of requests that are sorted by block address and issued together is far faster than the same requests issued in random order. An SSD's performance depends on the parallelism and the write pattern: a sequence of requests that are issued concurrently and written sequentially is faster than the same requests issued one at a time in random order. The modern I/O subsystem tries to satisfy both: it merges and sorts requests (which helps the disk), and it issues them concurrently and with large sequential writes (which helps the SSD). The algorithms that balance the two are the subject of Lesson 11.5.

4.12 Common misconceptions

"The processor talks to the device."

The processor talks to the device's controller, which talks to the device. The controller is the device's interface: it accepts the processor's commands, operates the device's physical components, and reports the results. The processor never sees the device's mechanical or electronic details.

"Memory-mapped I/O is just memory."

A memory-mapped device register is addressed like memory but does not behave like it. Reads and writes may have side effects, and two reads of the same register may return different values. The driver must use volatile accesses (or the kernel's accessor functions) to prevent the compiler and the processor from reordering, combining, or eliminating the accesses.

"Interrupts are always better than polling."

Interrupts save the processor's time when the expected wait is long, but they have overhead — typically a few hundred to a few thousand cycles per interrupt. For a very fast device (an NVMe SSD, a 100 Gb/s network adapter), the interrupt overhead can exceed the transfer's cost, and polling is faster. Modern high-performance drivers use polling for exactly this reason.

"DMA is always faster than programmed I/O."

DMA frees the processor, but it has a setup cost: the driver must allocate a buffer, translate its address, and program the DMA controller. For a small transfer, the setup cost can exceed the transfer's cost, and programmed I/O (or interrupt-driven I/O) is faster. DMA is the right choice for large transfers, not for all transfers.

"The bus is just a set of wires."

A modern bus is a protocol: PCI Express, SATA, SAS, and NVMe each define the physical layer, the packet format, the addressing, the flow control, and the error handling. The protocol determines the bus's bandwidth, its latency, and its features (hot-plug, message-signaled interrupts, multiple queues). The bus's protocol is as important as its wires.

"A device node is the device."

A device node (/dev/sda) is a name in the file system that the kernel maps to a device. The mapping is dynamic: the same physical device can appear under different names after a reboot, and the same name can refer to different devices. Modern systems use persistent identifiers (/dev/disk/by-uuid, /dev/disk/by-id) to avoid the ambiguity.

"SSDs have no performance characteristics to manage."

SSDs have no seek time, but they have write amplification, garbage collection, wear leveling, and parallelism. Their performance depends on the write pattern, the queue depth, and the alignment of the accesses. The operating system's I/O patterns and the file system's allocation policies affect the SSD's performance and lifetime, and the modern stack is designed with those effects in mind.

"The firmware's enumeration is final."

The firmware enumerates the buses during boot, but the kernel re-enumerates them during its initialization and after hot-plug events. A device that the firmware did not report can still be discovered by the kernel, and a device that is hot-plugged after boot is enumerated by the kernel's hot-plug machinery. The kernel's enumeration is authoritative.

05 Worked example

The following example traces a disk read through the I/O subsystem's layers, compares the cost of the three data-transfer mechanisms, and shows how the bus and the device's characteristics determine the transfer's performance.

Tracing a disk read

Consider a program that reads 64 KB from a file. The file system's block size is 4 KB, so the read spans 16 blocks. The blocks are allocated contiguously on the disk, and the disk's geometry is: 7,200 RPM, average seek time 8 ms, sequential bandwidth 200 MB/s.

The read passes through the following layers:

The application calls read(fd, buf, 65536). The kernel's VFS layer receives the call, looks up the file's inode (from the dcache, if the file has been accessed recently), and determines that the read spans blocks 2,048 through 2,063.
The VFS checks the page cache. If the blocks are already in the page cache (because a previous read or a readahead fetched them), the read is served from memory and the device is not touched. If the blocks are not cached, the VFS submits a read request to the block layer.
The block layer merges and schedules the request. The 16 blocks are contiguous, so the block layer merges them into a single request. The request is placed in the device's request queue, where the I/O scheduler (Lesson 11.5) may reorder it relative to other requests.
The device driver programs the controller. The driver translates the request — "read 64 KB starting at logical block 2,048" — into the controller's command format. It allocates a DMA buffer (or uses a scatter-gather list), translates the buffer's address, and writes the command, the address, and the count to the controller's registers.
The controller executes the command. The controller seeks the heads to the correct cylinder, waits for the platter's rotation to bring the first sector under the heads, and reads the 16 blocks into the DMA buffer. The controller's DMA engine writes the data to memory without the processor's involvement.
The controller interrupts. When the transfer is complete, the controller raises an interrupt. The driver's handler checks the status register, confirms the transfer's success, and signals the block layer.
The block layer completes the request. The data is now in the page cache. The VFS copies it to the application's buffer (or, if the application used mmap, maps it), and the read system call returns.

The trace shows the layering in action: the application's request is translated at each layer, and the hardware's details are hidden behind the driver's interface. The application does not know that the read required a seek, a rotational delay, and a DMA transfer; it sees only the bytes.

The transfer's cost

The read's cost has four components:

Component	Time	Notes
Seek	8 ms	The average seek time; the dominant cost for a random read
Rotational delay	4.2 ms	Half a rotation at 7,200 RPM
Transfer	0.32 ms	64 KB at 200 MB/s
Overhead	~0.05 ms	Driver, interrupt, and kernel overhead
Total	~12.6 ms	Dominated by the mechanical delays

The mechanical delays — the seek and the rotational delay — account for over 95% of the read's cost. The transfer itself is a small fraction. The observation is the reason the I/O subsystem's design is dominated by the goal of reducing the number of mechanical delays: the block layer merges contiguous requests (so one seek serves many blocks), the I/O scheduler sorts the requests (so the disk's heads move in one direction rather than back and forth), and the file system's allocation policy places a file's blocks contiguously (so a large read requires one seek rather than many).

Comparing the transfer mechanisms

Suppose the disk could not use DMA and the driver had to use programmed I/O or interrupt-driven I/O. The cost of transferring 64 KB would be:

Mechanism	Transfers	CPU time	Notes
Programmed I/O	8,192 (4-byte words)	~8 ms	The CPU is busy for the entire transfer
Interrupt-driven	16 (one per block)	~0.5 ms	The CPU is interrupted once per block
DMA	1 (one setup)	~0.01 ms	The CPU is involved only at the setup and the completion

The table is illustrative — the actual numbers depend on the processor, the bus, and the device — but the pattern is what matters. Programmed I/O consumes the processor for the entire transfer. Interrupt-driven I/O reduces the processor's involvement to one interrupt per block, but the interrupts still cost time and the CPU must move the data. DMA reduces the processor's involvement to a single setup and a single completion interrupt, and the data moves without the processor's attention.

The comparison also shows why DMA's setup cost matters. For a 64 KB transfer, the setup is negligible; for a 64-byte transfer, the setup may exceed the transfer's cost, and programmed I/O is the better choice. The driver's decision to use DMA or programmed I/O therefore depends on the transfer's size, and many drivers use a threshold: transfers smaller than a few hundred bytes use programmed I/O, and larger transfers use DMA.

The bus's effect

The bus determines the transfer's ceiling. Suppose the disk's sequential bandwidth is 200 MB/s and the bus is SATA at 6 Gb/s (about 600 MB/s after encoding). The bus is not the bottleneck: the disk can transfer at 200 MB/s, and the bus can carry three times that rate. If the disk were replaced by an NVMe SSD with a sequential bandwidth of 7 GB/s, the SATA bus would become the bottleneck: the SSD could transfer at 7 GB/s, but SATA would limit it to 600 MB/s. The SSD's full bandwidth requires an NVMe connection, which runs over PCIe and can carry tens of gigabytes per second.

The observation is the reason the storage bus hierarchy matters. A storage device's performance is limited by the slower of its own bandwidth and its bus's bandwidth, and the modern high-performance devices (NVMe SSDs, 100 Gb/s network adapters) require the high-speed buses (PCIe) to reach their potential. The operating system's drivers must be aware of the bus's characteristics, and the kernel's device model must expose them (through sysfs and the device's attributes) so that the administrator can diagnose the bottleneck.

Inspecting the I/O hardware on a live system

The Linux tools let an administrator inspect the system's I/O hardware: the buses, the devices, the interrupts, and the storage devices' characteristics.

# List the PCI devices and their buses
lspci -tv

# List the USB devices
lsusb -t

# List the block devices and their properties
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,ROTA,MODEL

# ROTA=1 means a rotational device (a hard disk);
# ROTA=0 means a non-rotational device (an SSD).

# Show the interrupt counts per device
cat /proc/interrupts

# Show the I/O port and memory regions in use
cat /proc/ioports | head -20
cat /proc/iomem | head -20

# Show the storage device's characteristics
sudo hdparm -I /dev/sda 2>/dev/null | head -30
sudo smartctl -i /dev/sda 2>/dev/null | head -20

The commands' outputs show the hardware's configuration: the PCI hierarchy, the USB devices, the block devices and whether they are rotational, the interrupt counts (which reveal which devices interrupt most frequently), the I/O port and memory regions (which reveal the memory-mapped and port-mapped devices), and the storage devices' capabilities (their model, their firmware, their supported transfer modes, and their SMART health).

06 Interactive activity

Activity · explore the I/O hardware on a live system

The goal of this activity is to use the Linux command line to inspect the system's I/O hardware, to observe the buses and the devices, and to measure the device's performance. Work through the steps in your course VM.

Inventory the system's devices. Use lspci, lsusb, and lsblk to list the devices.
```
# List the PCI devices
lspci

# List the USB devices
lsusb

# List the block devices
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,ROTA,MODEL

# List the CPU and its features
lscpu | head -20
```
How many PCI devices does the system have? How many USB devices? How many block devices? Which block devices are rotational and which are non-rotational? Write down what you find.
Examine the bus topology. Use lspci -tv and lsusb -t to see the buses' tree structure.
```
# Show the PCI tree
lspci -tv

# Show the USB tree
lsusb -t

# Show the block devices' bus connections
lsblk -o NAME,SIZE,TYPE,TRAN,MODEL

# The TRAN column shows the transport (sata, nvme, usb, etc.).
```
What buses does the system use? Which devices are on the PCIe bus? Which are on USB? Which storage devices use SATA, and which use NVMe? Explain how the topology reflects the devices' bandwidth requirements.
Examine the interrupt distribution. Use /proc/interrupts to see which devices interrupt the processor and how frequently.
```
# Show the interrupt counts
cat /proc/interrupts

# Run a disk-heavy workload
dd if=/dev/sda of=/dev/null bs=1M count=1000 2>/dev/null

# Show the interrupt counts again
cat /proc/interrupts
```
Which devices have the highest interrupt counts? How did the counts change after the disk workload? What does the change tell you about the devices' interrupt behaviour? Are the storage interrupts message-signaled (they appear with names like "nvme0q1" or "ahci") or legacy?
Examine the memory-mapped and port-mapped regions. Use /proc/iomem and /proc/ioports to see the devices' address regions.
```
# Show the I/O memory regions
cat /proc/iomem | head -40

# Show the I/O port regions
cat /proc/ioports | head -40

# Look for the devices' BARs and the port ranges.
# The memory regions are the memory-mapped devices;
# the port regions are the port-mapped devices.
```
How many memory-mapped regions are there? How many port-mapped regions? Which devices use which? What does the distribution tell you about the prevalence of memory-mapped I/O on a modern system?

Measure a storage device's performance. Use hdparm or dd to measure the device's sequential and random throughput.

# Measure the sequential read throughput (requires root)
sudo hdparm -tT /dev/sda

# Measure with dd (create a 1 GB file first)
dd if=/dev/zero of=/tmp/test.bin bs=1M count=1024 2>/dev/null
sync
echo 3 | sudo tee /proc/sys/vm/drop_caches

# Sequential read
time dd if=/tmp/test.bin of=/dev/null bs=1M

# Random read (with a small block size and a seek pattern)
time dd if=/tmp/test.bin of=/dev/null bs=4k count=1000 iflag=direct

# Clean up
rm /tmp/test.bin

What is the device's sequential throughput? What is its random throughput? Is the difference large (suggesting a hard disk) or small (suggesting an SSD)? How does the result compare to the theoretical bandwidth of the device's bus?

Compare the storage devices. If the system has more than one storage device, compare their performance and their characteristics.
```
# List the storage devices and their types
lsblk -d -o NAME,SIZE,ROTA,TRAN,MODEL

# Measure each device's read throughput
for dev in $(lsblk -dn -o NAME); do
    echo "=== /dev/$dev ==="
    sudo hdparm -t /dev/$dev 2>/dev/null
done
```
Which devices are fastest? Which are rotational, and which are solid-state? How does the performance correlate with the device type and the bus? Explain the correlation in terms of the devices' characteristics from the lesson.
Observe the device nodes. Use ls -l /dev and /sys/block to see how the kernel presents the devices.
```
# List the block device nodes
ls -l /dev/sd* /dev/nvme* 2>/dev/null

# Show the block devices' sysfs entries
ls /sys/block/

# Show a device's properties
cat /sys/block/sda/queue/rotational
cat /sys/block/sda/queue/scheduler
cat /sys/block/sda/queue/nr_requests
cat /sys/block/sda/size

# Show the persistent identifiers
ls -l /dev/disk/by-id/ 2>/dev/null | head -10
```
What are the major and minor numbers of the block devices? Is the device rotational (1) or non-rotational (0)? What I/O scheduler is in use? What is the device's size in 512-byte sectors? What persistent identifiers are available?
Read the man pages. Read the man pages that document the I/O hardware tools.
```
man 8 lspci
man 8 lsusb
man 1 lsblk
man 1 lscpu
man 5 proc
man 8 hdparm
man 8 smartctl
```
For each, write a sentence describing its main contribution to your understanding of the I/O hardware. The man 8 lspci page documents the PCI device listing; the man 5 proc page documents the /proc interfaces; the man 8 hdparm page documents the storage device parameter tool.

Reflect: the I/O subsystem is invisible when it works and infuriating when it does not. The user plugs in a device and it works; the application reads a file and the bytes appear. The invisibility is the result of the layering, the bus protocols, the enumeration, and the three transfer mechanisms — decades of engineering that the user never sees. Which of the mechanisms from this lesson matter most for the devices you use? What would you change about the I/O subsystem's design if you could? How would you explain the difference between a device and its controller to someone who has never thought about it? What is the right balance between the hardware's capabilities and the operating system's abstractions?

07 Knowledge check

Check your understanding

Question 1

Describe the I/O subsystem's layers. What is each layer's responsibility, and why is the layering important?

Show answer

The I/O subsystem is organized as a stack of layers. The application issues the I/O request through the operating system's interface — the system calls read, write, open, close, ioctl — or a library that wraps them. The device-independent I/O software is the kernel layer that provides the common interface: the file system's VFS, the buffering and caching, the error handling, and the device naming. It translates the application's request ("read 4 KB from offset 1 MB of this file") into a request on a specific device ("read blocks 2,048 through 2,055 of this partition"). The device driver is the kernel module that knows how to operate a specific device or class of devices. It translates the device-independent request into the sequence of register operations and memory transfers that the device's controller requires, and it handles the device's interrupts and errors. The device controller is the hardware that drives the device. It receives commands from the processor through its registers, executes them by operating the device's mechanical or electronic components, and reports their completion through its registers and its interrupts. The device is the mechanical or electronic component that performs the physical operation. The layering is important because it lets each layer be replaced without disturbing the others: a new device requires a new driver, not a new file system; a new file system requires a new VFS implementation, not a new driver. The application does not know which device holds its data; the file system does not know how the controller works; the driver does not know why the data is being read. Each layer translates between the layer above and the layer below, and the translation hides the details that the other layers do not need.

Question 2

Compare port-mapped and memory-mapped I/O. What are the advantages and disadvantages of each, and which is used on modern systems?

Show answer

Port-mapped I/O (isolated I/O) gives the device's registers their own address space, separate from memory. The processor uses special instructions (on x86, IN and OUT) to read and write the port space, and the bus carries a signal that distinguishes a port access from a memory access. The port space is small — 65,536 ports on x86 — and the device's registers occupy a few ports within it. Memory-mapped I/O maps the device's registers into the processor's memory address space. A range of memory addresses is reserved for the device's registers, and the processor uses its ordinary load and store instructions to read and write them. The memory controller or the bus's address decoder routes the accesses to the device. The two schemes have different trade-offs. Memory-mapped I/O is simpler for the software: the driver can use ordinary load and store instructions, and the compiler can generate them without special intrinsics. It is also faster on modern hardware, because the processor's load and store instructions are optimized for the memory hierarchy and can be pipelined and reordered, while the special I/O instructions of port-mapped I/O are serializing on many processors. Its disadvantage is that it consumes memory address space, although on a 64-bit processor the loss is irrelevant. Port-mapped I/O is isolated: the device's registers cannot be confused with memory, and a wild pointer cannot accidentally write to a device's control register. The isolation is a safety property, but it is not a strong one, because a driver can still write to a device's ports deliberately. Modern systems use memory-mapped I/O almost exclusively: PCI Express devices are memory-mapped by design, and the port-mapped space is reserved for legacy devices and for the processor's own control registers. The Linux kernel's drivers use ioremap to map a device's registers into the kernel's address space and readl/writel to access them. The driver must be careful, however: a memory-mapped register looks like memory but does not behave like it, because reads and writes may have side effects and two reads may return different values. The driver must use volatile accesses (or the kernel's accessor functions, which include the necessary barriers) to prevent the compiler and the processor from reordering, combining, or eliminating the accesses.

Question 3

Compare programmed I/O, interrupt-driven I/O, and direct memory access. What is each mechanism's cost, and when is each appropriate?

Show answer

The three mechanisms differ in how much of the transfer the processor performs. Programmed I/O transfers each byte or word with the processor's own load and store instructions. It is usually combined with polling: the processor issues a command and then repeatedly reads the device's status register until the device is ready. The mechanism's advantage is its simplicity — no interrupts, no interrupt handler, no interrupt controller — and its predictability. Its disadvantage is its waste: the processor spends the entire transfer in the polling loop, and the time it spends is proportional to the device's latency rather than to the amount of data transferred. It is appropriate for very fast devices (where the interrupt overhead would exceed the transfer's cost), for short transfers, and for idle processors. Interrupt-driven I/O lets the processor issue a command and continue with its other tasks. When the device completes the command, its controller raises an interrupt, and the processor suspends its current task and jumps to the device's interrupt handler. The handler reads the status register, transfers any remaining data, and returns. The mechanism's advantage is that the processor is free during the device's operation. Its disadvantages are the interrupt's overhead — saving the processor's state, entering the handler, identifying the device, and returning typically take a few hundred to a few thousand cycles — and the fact that the processor still moves the data. It is appropriate for slow and medium-speed devices (keyboards, mice, serial ports, network adapters at moderate rates) and for small transfers. Direct memory access (DMA) transfers data between a device and memory without the processor's per-byte involvement. The processor sets up the transfer by writing the memory address, the byte count, and the direction to a DMA controller, and the controller performs the transfer and raises an interrupt when it is complete. The mechanism's advantage is that the processor is free for the whole transfer, not just for the wait; the cost is a small setup and a single completion interrupt. Its complications are the setup cost (which can exceed the transfer's cost for small transfers) and the physical-address problem (the buffer's virtual pages may be scattered across physical memory, which the operating system handles with contiguous allocation, scatter-gather DMA, or an IOMMU). It is appropriate for large transfers on high-bandwidth devices: disks, SSDs, and high-speed network adapters. The general rule is that programmed I/O is appropriate when the expected wait is short and the processor has no other work; interrupts are appropriate when the wait is long or the processor is busy; and DMA is appropriate when the transfer is large enough that the setup cost is amortized.

Question 4

Describe the bus hierarchy on a modern system. What are the levels, what devices are connected to each, and why is the hierarchy used instead of a single bus?

Show answer

The bus is the set of wires or serial links that carry data, addresses, and control signals between the processor, memory, and the devices' controllers. Modern computers use a hierarchy of buses rather than a single bus, because a single bus cannot satisfy all the devices' requirements. The hierarchy has three levels. The memory bus connects the processor and main memory. It is the fastest bus in the system — tens of gigabytes per second — and it is optimized for the processor's memory accesses. Devices are not connected to it directly; they reach memory through a bridge. The general-purpose I/O bus connects the high-bandwidth devices — graphics adapters, storage controllers, network adapters — to the processor and memory. On modern systems this is PCI Express (PCIe), a high-speed serial bus organized as a set of lanes; a device can use one, four, sixteen, or more lanes depending on its bandwidth requirement. The low-speed expansion buses connect the slow devices — keyboards, mice, printers, external storage — to the system. USB is the standard for external devices; SATA and SAS are the buses for internal storage devices; NVMe is the protocol that runs over PCIe for high-performance SSDs. The hierarchy is used because it matches each device to a bus that suits its bandwidth and cost: a keyboard does not need a PCIe ×16 connection, and a graphics adapter cannot use USB. The bridges between the buses translate the protocols, so a device on a low-speed bus can still be addressed by the processor and can still interrupt it. The translation's cost is the reason the hierarchy is arranged with the fastest devices closest to the processor: a device on a slower bus pays more latency per access. For storage devices, the hierarchy matters because the bus's bandwidth and latency determine the device's performance ceiling: a storage device connected to a SATA port cannot exceed SATA's bandwidth, no matter how fast the device is, while an NVMe SSD can use the PCIe bus's full bandwidth.

Question 5

Describe a hard disk's geometry and a solid-state drive's characteristics. How do the differences affect the I/O software's design?

Show answer

A hard disk's physical components are the platters (rotating disks coated with magnetic material), the heads (the read/write heads, one per platter surface), the tracks (the concentric circles on a platter's surface), the sectors (the subdivisions of a track, 512 bytes classically and 4,096 bytes in the modern Advanced Format), and the cylinders (the set of tracks at the same radial position on all the platters' surfaces). Reading or writing a sector requires three operations: a seek (moving the heads to the correct cylinder, a few milliseconds), a rotational delay (waiting for the platter's rotation to bring the sector under the head, about 4.2 ms on average at 7,200 RPM), and a transfer (a few microseconds per sector). The mechanical delays dominate: a random read costs a few milliseconds, of which the transfer is a tiny fraction, while a sequential read costs one seek and one rotational delay for the first sector and then only the transfer time for the rest. A solid-state drive stores data in flash memory and has different characteristics: it has no mechanical delay, so its access time is dominated by the flash memory's read latency (tens of microseconds) and the difference between sequential and random access is far smaller; its flash memory cannot be overwritten in place, so a block must be erased before it can be written, and the erase operation works on a larger unit than the read/write unit; each flash cell can be erased and rewritten only a limited number of times, so the drive's controller performs wear leveling; the controller's operations cause write amplification (more physical writes than the operating system requested); and the drive contains many flash chips that the controller can access in parallel, which is the source of the drive's high throughput. The differences affect the I/O software's design. On a hard disk, the dominant goal is to reduce the number of mechanical delays: the block layer merges contiguous requests so one seek serves many blocks, the I/O scheduler sorts the requests so the heads move in one direction, and the file system's allocation policy places a file's blocks contiguously. On an SSD, the seek reduction is less important, and the dominant goals are to maximize parallelism and to minimize write amplification: the NVMe protocol's queue-based interface lets the operating system submit thousands of commands at once, the operating system tells the drive which blocks are free (the TRIM command) so the controller does not copy them, and the file systems' allocation policies try to write sequentially and in aligned units. The flash-specific file systems (F2FS) are designed around the SSD's constraints, and the general-purpose file systems' flash-friendly modes cooperate with the drive's controller rather than fighting it.

08 Unit discussion

Discussion prompt

The I/O subsystem is the operating system's largest and most heterogeneous component, and its history is a history of accommodation. The first computers had no I/O subsystem to speak of: the program read the paper tape or the punched cards directly, and the processor was involved in every transfer. The interrupt was invented to free the processor during the device's operation; DMA was invented to free it during the transfer itself; the bus hierarchy was invented to match the devices' bandwidth to the buses' capabilities; and the device model was invented to make the system self-configuring. Each mechanism was a response to a limitation of the previous one, and each was shaped by the technology of its time.

The trend is toward abstraction and autonomy. The processor's involvement in I/O has steadily decreased: from programmed I/O (the processor moves every byte) to interrupt-driven I/O (the processor moves the data but waits) to DMA (the processor sets up and waits) to modern devices with their own processors, their own memory, and their own scheduling. A modern NVMe SSD contains more computing power than the computers of the 1970s, and it makes its own decisions about how to schedule the requests it receives. The operating system's role is shifting from driving the device to advising it: the driver submits the requests, and the device's controller decides how to execute them.

The shift raises a question about the division of labour. If the device's controller is smart enough to schedule the requests, should the operating system still try to sort and merge them? If the device's flash translation layer knows the flash chips' state, should the file system still try to allocate contiguously? If the device can reorder the requests, does the I/O scheduler still have a role? The modern answers are mixed: the NVMe protocol's multiple queues assume that the operating system will submit many concurrent requests, but the device's controller schedules them; the file system's allocation policies still try to write sequentially, but the drive's wear leveling may scatter the writes anyway; the I/O scheduler's role has shrunk on SSDs, but it has not disappeared.

Where should the boundary between the operating system and the device's controller be? Should the operating system expose the device's details to the application, or should it hide them behind the uniform interface? What is the right balance between the hardware's autonomy and the operating system's control? Post your response to the Unit 11 discussion forum and reply to at least two classmates. Supports CLO1, CLO5, and CLO6.

09 Practice exercise

This ungraded exercise consolidates the lesson and prepares you for Lesson 11.2. Aim to spend 60–90 minutes on it.

Trace an I/O request. A program writes 8 KB to a file. Trace the write through the I/O subsystem's layers, from the application's write system call to the device's controller. For each layer, describe:
- What the layer receives from the layer above.
- What the layer does with it.
- What the layer passes to the layer below.
Assume the file's blocks are not in the page cache and the device uses DMA.
Compare the transfer mechanisms. Write a short paragraph (200–250 words) comparing programmed I/O, interrupt-driven I/O, and direct memory access. Your paragraph should address: (a) how much of the transfer the processor performs; (b) the mechanism's cost; (c) the mechanism's appropriate use cases; (d) the complications (interrupt overhead, DMA setup, the physical-address problem). Conclude with a rule of thumb for choosing among the three.

Measure the interrupt rate. Use /proc/interrupts to measure the interrupt rate of the storage device under different workloads.

# Record the interrupts before the workload
grep -E 'ahci|nvme|sd' /proc/interrupts

# Run a sequential read
dd if=/dev/sda of=/dev/null bs=1M count=1000 2>/dev/null

# Record the interrupts after
grep -E 'ahci|nvme|sd' /proc/interrupts

# Run a random read
dd if=/dev/sda of=/dev/null bs=4k count=1000 iflag=direct 2>/dev/null

# Record the interrupts after
grep -E 'ahci|nvme|sd' /proc/interrupts

How many interrupts did each workload generate? Which workload generated more interrupts per byte? Explain the difference in terms of the transfer sizes and the device's interrupt behaviour.

Compare the storage devices' throughput. If the system has both a hard disk and an SSD (or two devices with different types), compare their sequential and random throughput.
```
# Identify the devices and their types
lsblk -d -o NAME,SIZE,ROTA,TRAN,MODEL

# Measure each device's sequential read
for dev in $(lsblk -dn -o NAME); do
    echo "=== /dev/$dev sequential ==="
    sudo hdparm -t /dev/$dev 2>/dev/null
done

# Measure each device's random read (if the device has
# a mounted file system, use a file on it)
# ...
```
What is the ratio of the sequential throughputs? What is the ratio of the random throughputs? Which device shows a larger difference between sequential and random access? Explain the difference in terms of the devices' mechanical characteristics.
Investigate a device's registers. On a system where you have the necessary permissions, use lspci -vv to examine a device's registers and its BAR.
```
# Show a device's configuration space and BARs
lspci -vv -s 00:1f.2 2>/dev/null | head -60

# The output shows the device's memory-mapped and
# port-mapped regions, its interrupt line, and its
# capabilities.
```
What memory-mapped regions does the device have? What port-mapped regions? What interrupt does it use? Does it support message-signaled interrupts (MSI or MSI-X)? Explain how the regions and the interrupt correspond to the lesson's concepts.
Design an I/O configuration. A server is being built for a database workload. The database performs many random reads (4 KB each) and many small writes, and it requires low latency and high throughput. The server has a PCIe 4.0 bus with 16 lanes available for storage. Design the storage configuration. Consider:
- Which storage devices would you use (hard disks, SATA SSDs, NVMe SSDs)?
- How many devices, and how would you connect them to the bus?
- How would you configure the redundancy?
- How would the device's characteristics (access time, write amplification, parallelism) affect the design?
- How would the operating system's I/O scheduler and the file system's allocation policy interact with the devices?
Justify your design.
Reflect. In your own words, explain why the I/O subsystem is layered, and describe the three data-transfer mechanisms and their roles. Then describe one situation in your own experience — a program you have written or a system you have used — where the I/O hardware's characteristics affected performance. What was the workload, and how would a different device or a different configuration have changed the outcome?

10 Summary

The I/O subsystem is the collection of hardware and software that connects the processor and memory to the I/O devices. It is organized as a stack of layers: the application, the device-independent I/O software, the device driver, the device controller, and the device.
A device is a piece of hardware that performs an I/O operation; a controller is the electronics that operates the device on the processor's behalf. The processor communicates with the controller through its registers: the data registers, the control registers, the status registers, and the interrupt-related registers.
Port-mapped I/O gives the device's registers their own address space, accessed with special instructions. Memory-mapped I/O maps the registers into the processor's memory address space, accessed with ordinary load and store instructions. Modern systems use memory-mapped I/O almost exclusively, but a memory-mapped register must be accessed with volatile semantics because its reads and writes may have side effects.
Programmed I/O transfers each byte or word with the processor's own instructions, and polling repeatedly reads the device's status register to detect completion. The mechanism is simple but keeps the processor busy for the entire transfer. It is appropriate for very fast devices, short transfers, and idle processors.
Interrupt-driven I/O lets the processor issue a command and continue with its other tasks; the device's controller raises an interrupt when the command completes. The mechanism frees the processor during the wait but pays an interrupt's overhead per transfer. It is appropriate for slow and medium-speed devices and for small transfers.
Direct memory access (DMA) transfers data between a device and memory without the processor's per-byte involvement. The processor sets up the transfer and the DMA controller performs it, raising one interrupt at the end. DMA's complications are its setup cost and the physical-address problem, which the operating system handles with contiguous allocation, scatter-gather DMA, or an IOMMU.
The bus carries the data, addresses, and control signals between the processor, memory, and the devices' controllers. Modern systems use a hierarchy of buses: the memory bus, the general-purpose I/O bus (PCI Express), and the low-speed expansion buses (USB, SATA, SAS). NVMe is the protocol that runs over PCIe for high-performance SSDs, and its queue-based interface is designed for flash storage's parallelism.
The kernel organizes the devices into classes: block devices (fixed-size blocks, random access, cached in the page cache), character devices (streams of bytes, no random access, unbuffered), and network devices (packets, addressed by network address, accessed through the socket API). The class determines the kernel's caching and buffering strategy.
Enumeration is the process of discovering the devices and assigning their addresses. It is a collaboration between the firmware, the buses' controllers, and the kernel: the firmware scans the buses during boot, the kernel re-scans them and matches the devices to drivers, and the kernel's device model creates the device nodes. The result is a self-configuring system.
A hard disk's geometry — platters, heads, tracks, sectors, cylinders — determines its performance: a read requires a seek, a rotational delay, and a transfer, and the mechanical delays dominate. A solid-state drive has no mechanical delay, but it has erase-before-write, wear, write amplification, and parallelism. The differences shape the I/O software: hard disks reward seek reduction, and SSDs reward parallelism and sequential writes.

Looking ahead

This lesson presented the I/O subsystem's hardware: the devices and controllers, the registers, the two addressing schemes, the three data-transfer mechanisms, the bus hierarchy, the device classes, enumeration, and the storage devices' characteristics. It referred several times to the interrupt — the signal that the device's controller raises when a command completes — and to the interrupt handler that the processor enters. Lesson 11.2 presents interrupts in detail: the interrupt mechanism, the interrupt controller, the interrupt vector table, the handler's structure, the difference between interrupts and exceptions, the prioritization and masking, the top-half and bottom-half split, and the modern alternatives (message-signaled interrupts, threaded interrupts). The interrupt is the mechanism that makes interrupt-driven I/O and DMA possible, and it is the bridge between the hardware of this lesson and the software of the next.

11 References

Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 12, I/O Systems.
Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapter 36, I/O Devices. Available online at ostep.org.
Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 5, Input/Output.
Love, R. Linux Kernel Development, 3rd ed., Chapter 14, Block Devices I/O, and Chapter 16, The Block I/O Layer.
Bovet, D. P., & Cesati, M. Understanding the Linux Kernel, 3rd ed. O'Reilly, 2005. Chapter 13, I/O Architecture and Device Drivers.
Kerrisk, M. The Linux Programming Interface. No Starch Press, 2010. Chapters 4 and 13.
Patterson, D. A., & Hennessy, J. L. Computer Organization and Design. The treatment of the I/O bus, the interrupt mechanism, and memory-mapped I/O.
Linux man-pages: man 4 tty, man 4 sd, man 4 null, man 4 zero, man 8 lspci, man 8 lsusb, man 1 lsblk, man 1 lscpu, man 5 proc, man 8 hdparm, man 8 smartctl, man 1 iostat.