Device Drivers

01 Introduction

Lesson 11.1 presented the I/O subsystem's layers and placed the device driver between the kernel's device-independent software and the device's controller. Lesson 11.2 presented the interrupt mechanism that the driver uses to learn of a device's completion, and Lesson 11.3 presented the DMA machinery that the driver uses to move the device's data. This lesson examines the driver itself: the kernel module that knows how to operate a specific device, and that bridges the kernel's abstractions and the device's registers.

The driver's role is translation. The kernel above the driver presents a uniform interface: the virtual file system's read, write, open, and close operations; the block layer's requests; the network stack's packets. The device below the driver presents a specific interface: a set of registers, a command format, a protocol for reporting completion, and a set of capabilities. The driver's job is to translate between the two. It receives a generic request — "read 4 KB from offset 1 MB" — and turns it into the device's specific sequence of register writes, DMA descriptors, and interrupt handling. It receives the device's completion — a status register's value, a DMA descriptor's completion — and turns it into the kernel's generic signal — a request's completion, a process's wakeup.

The translation is the reason drivers dominate a modern kernel's code. A device's interface is not standardized: two storage controllers may implement the same protocol (AHCI, NVMe) but with different register layouts, different command formats, and different quirks; two network adapters may implement the same Ethernet standard but with different descriptor formats and different offload capabilities. Each device needs its own driver (or, more precisely, each device or family of devices needs a driver that knows its details), and the drivers together account for the majority of the Linux kernel's source lines. The observation is why the kernel provides so much infrastructure — the device model, the bus types, the DMA API, the interrupt API, the locking primitives — to make the drivers' work manageable.

This lesson presents the device driver. It defines the driver and its role. It presents the kernel module mechanism — the loadable modules that let the kernel support new devices without being recompiled — and the driver's structure: its initialization, its probe and remove functions, and its file operations. It presents the three device classes' interfaces: the character device's file_operations, the block device's request queue and the multi-queue block layer, and the network device's net_device and socket buffer. It presents the driver's use of interrupts and DMA, and the Linux device model — the bus, device, driver, and class abstractions that let the kernel match devices to drivers and expose them in sysfs. It presents the practical concerns of driver development: the locking rules, the sleeping rules, the error handling, and the debugging tools. It presents the user-space alternatives — FUSE, UIO, VFIO, DPDK — that move the driver out of the kernel, and it closes with the design trade-offs that shape the driver's structure.

02 Learning objectives

This lesson contributes to U11.4 and U11.5, and to course learning outcomes CLO1, CLO6, and CLO8.

03 Required reading

• Operating System Concepts — Chapter 12, I/O Systems, Section 12.5, I/O Requests to Hardware Operations, and Section 12.6, STREAMS. The standard treatment of the I/O request's path through the kernel and the driver's role.
• OSTEP — Chapter 36, I/O Devices. OSTEP's discussion of the device's interface and the operating system's role in managing it is the clearest available account, and it is the basis for the worked example in this lesson.
• Tanenbaum & Bos — Modern Operating Systems, Chapter 5, Sections 5.2–5.4. A complementary treatment that develops the device driver's structure and its interaction with the kernel from first principles.
• Corbet, J., Rubini, A., & Kroah-Hartman, G. — Linux Device Drivers, 3rd ed. O'Reilly, 2005. The definitive guide to Linux driver development. Chapters 1–3 (the driver's structure and the character device interface), Chapters 4–6 (locking, advanced character drivers, and the ioctl interface), Chapters 10–12 (interrupt handling, the kernel data types, and the PCI interface), and Chapter 14 (the Linux device model).
• Love, R. — Linux Kernel Development, 3rd ed., Chapter 14, Block Devices I/O, and Chapter 16, The Block I/O Layer. The Linux kernel's block-device structures, the request queue, and the multi-queue block layer.
• Bovet, D. P., & Cesati, M. — Understanding the Linux Kernel, 3rd ed. O'Reilly, 2005. Chapter 13, I/O Architecture and Device Drivers. The Linux kernel's device model, bus structures, and the driver's registration.
• Kerrisk, M. — The Linux Programming Interface, Chapter 14, File Systems, and Chapter 15, File Attributes. The application's view of the device files and the ioctl interface.
• Linux man pages: man 2 ioctl, man 4 mem, man 4 null, man 4 zero, man 4 random, man 4 tty, man 4 sd, man 5 sysfs, man 8 insmod, man 8 rmmod, man 8 modprobe, man 8 lsmod, man 8 modinfo, man 1 udevadm, man 8 lspci, man 8 lkddb.

04 Core content

4.1 What a device driver is

A device driver is a piece of kernel code that knows how to operate a specific device (or a family of devices) and that presents the device to the rest of the kernel through one of the kernel's standardized interfaces. The driver is the kernel's translation layer: it receives the generic requests that the kernel's upper layers produce, and it turns them into the device-specific operations that the device's controller requires.

The driver's position is between two very different interfaces:

• The interface above the driver is the kernel's abstraction. For a storage device, the interface is the block layer's request queue: the driver receives a request that describes a read or a write of a range of blocks. For a character device, the interface is the file operations: the driver receives open, read, write, ioctl, and close calls from the kernel's VFS layer. For a network device, the interface is the network stack's packet interface: the driver receives a packet to transmit or delivers a received packet. The interface is the same for all devices of the same class, so the kernel's upper layers do not need to know which device they are using.
• The interface below the driver is the device's hardware. For a storage controller, the interface is a set of registers: a command register, a status register, a DMA address register, and so on. For a network adapter, the interface is a set of descriptor rings and control registers. For a USB device, the interface is a set of endpoints and a protocol. The interface is different for every device, so the driver must know the device's details.

The driver's translation is not merely mechanical. It must handle the device's quirks (a controller that requires a specific command order, an adapter that misreports its capabilities), its error conditions (a transfer that fails, a device that disappears), its timing constraints (a register that must be read within a specific window), and its concurrency (a device that can process several commands at once). It must do all of this while obeying the kernel's rules about locking, sleeping, and memory allocation. The driver is therefore one of the kernel's most demanding components to write, and its quality determines the system's reliability: a buggy driver can crash the kernel, corrupt data, or hang the system, and driver bugs are the source of a large fraction of the kernel's reported crashes.

4.2 Kernel modules and loadable modules

A modern kernel does not include every driver in its base image. Instead, it uses loadable kernel modules: pieces of kernel code that can be loaded and unloaded at runtime, without recompiling the kernel or rebooting the system. A module is a compiled object file (a .ko file) that contains the driver's code and a small amount of metadata that describes what the module does and what it needs.

The module mechanism has three advantages:

• Flexibility. A system can support a device by loading the device's module, without recompiling the kernel. The module can be built for the running kernel's version, so a system can be updated without rebuilding its drivers.
• Memory efficiency. A module is loaded only when it is needed. A system with a hundred possible devices loads only the modules for the devices it actually has, so the kernel's memory footprint is smaller.
• Development speed. A driver can be developed, tested, and debugged as a module, without the cycle of building and booting a new kernel. The module can be unloaded and reloaded after each change, so the development cycle is seconds rather than minutes.

The module mechanism's core is the kernel's module loader. The loader:

1. Reads the module's metadata. The module's .modinfo section describes the module's name, its author, its license, its parameters, and (most importantly) the devices it supports. The metadata is what lets the kernel match a device to a module.
2. Resolves the module's dependencies. A module may use symbols that other modules export (a driver may use the USB subsystem's functions, a file system may use the block layer's functions). The loader reads the module's undefined symbols, finds the modules that define them, and loads those modules first.
3. Relocates the module's code. The module's code is compiled as position-independent (or with relocations), and the loader adjusts the addresses to the kernel's memory layout.
4. Calls the module's initialization function. The module's module_init function is called; it registers the driver with the kernel's device model and performs the module's other setup.

The module's unload is the reverse: the kernel calls the module's module_exit function, which unregisters the driver and releases the module's resources; then the loader frees the module's memory. The unload can fail if the module is in use (a file system is mounted, a device is open), so the kernel maintains a reference count for each module and refuses to unload a module whose count is nonzero.

The module mechanism's interface to user space is the modprobe, insmod, rmmod, lsmod, and modinfo commands, and the /sys/module/ directory. The udev daemon uses the module's metadata to load the modules for the devices that appear, so the system is self-configuring: the user plugs in a device, udev recognizes it, and the kernel loads the appropriate module. The mechanism is one of the reasons Linux supports such a wide range of hardware: the drivers are modules, and the distribution ships modules for thousands of devices, loading only the ones the system needs.

4.3 The driver's structure

A Linux driver has a recognizable structure, with four parts:

Initialization and cleanup

The module_init function runs when the module is loaded (or when the kernel boots, if the driver is built into the kernel). It registers the driver with the kernel's device model, allocates the driver's global data structures, and performs any global setup. The module_exit function runs when the module is unloaded; it reverses the initialization, unregisters the driver, and frees the global structures. The two functions are the module's entry and exit points, and their structure is defined by the module_init and module_exit macros.

Probe and remove

The probe function runs when the kernel matches the driver to a device. It is the driver's per-device initialization: it enables the device's registers (mapping them into the kernel's address space), allocates the device's per-device data structures, requests the device's interrupts and DMA channels, registers the device with the appropriate subsystem (the block layer, the network stack, the character device layer), and initializes the device's hardware. The remove function runs when the device is removed (or when the module is unloaded); it reverses the probe's work and releases the device's resources. The probe and remove functions are the driver's per-device entry and exit points, and their structure is defined by the bus type (the PCI, USB, or platform bus's probe and remove signatures).

The interface implementation

The driver implements the interface that its device class requires. A character driver implements the file_operations structure's methods (open, read, write, ioctl, release, and so on). A block driver implements the block layer's interface (the queue_rq method of the blk_mq_ops structure). A network driver implements the net_device_ops structure's methods (ndo_start_xmit, ndo_open, ndo_stop, and so on). The interface is what the kernel's upper layers call, and its methods are the driver's actual work.

The interrupt and deferred-work handlers

If the device generates interrupts, the driver implements the interrupt handler (the top half) and the deferred-work handler (the bottom half, as a softirq, a tasklet, or a workqueue; Lesson 11.2). The top half acknowledges the device and schedules the bottom half; the bottom half processes the completed work. The driver's interrupt handling is where the device's asynchronous behaviour is translated into the kernel's request completion.

The four parts' interaction is the driver's lifecycle. The module's initialization registers the driver; the kernel's device model matches the driver to a device and calls the probe; the probe sets up the device and registers the driver's interface; the kernel's upper layers call the interface's methods; the device's interrupts and DMA completions drive the driver's handlers; the device's removal (or the module's unload) calls the remove; the module's cleanup unregisters the driver. The lifecycle is the same for every driver, regardless of the device's class, and its structure is the kernel's contribution to making the drivers' work manageable.

4.4 Character device drivers

A character device is a device that transfers data as a stream of bytes and does not support random access. Its interface is the file_operations structure, a set of function pointers that the kernel's VFS layer calls when an application performs a file operation on the device's node.

The file_operations structure's most important methods are:

open

Called when an application opens the device's node. The method initializes the device's per-open state (or the device's global state, if the device can be opened only once), checks the requested access mode (read, write, or both), and returns a file descriptor. The method is the device's initialization point for each open.

read

Called when an application reads from the device. The method transfers data from the device to the application's buffer, using the kernel's copy_to_user function (which validates the user-space address and performs the copy with the necessary protections). The method returns the number of bytes transferred, or a negative error code.

write

Called when an application writes to the device. The method transfers data from the application's buffer to the device, using the kernel's copy_from_user function. The method returns the number of bytes transferred, or a negative error code.

ioctl

Called when an application performs a device- specific operation. The method's arguments are a command number and an optional argument; the command number's format is defined by the kernel (the _IO, _IOR, _IOW, and _IOWR macros), and the method's implementation interprets the command. The ioctl interface is the device's escape hatch for the operations that the read and write interface cannot express (setting the device's mode, querying its status, performing a control operation).

release

Called when the last reference to the device's open file is closed. The method releases the per-open state and (if the device is exclusive) allows the next open.

mmap

Called when an application maps the device's memory into its address space (through the mmap system call). The method sets up the mapping, typically by using the kernel's remap_pfn_range function to map the device's physical pages (the device's memory-mapped registers or its frame buffer) into the application's address space. The method is used by the devices that expose their memory to applications: graphics cards, frame buffers, high-speed network adapters, and the devices that support user-space drivers.

The character device's interface is deliberately simple, and its simplicity is its strength: an application can use the device with the same open, read, write, and close calls that it uses for a file, and the device's driver does not need to implement any special protocol. The interface's simplicity is also its limitation: the character device's operations are a stream, so the device cannot be accessed randomly, and the ioctl interface (which the driver must implement for the device-specific operations) is notoriously difficult to design well (its commands are not standardized, its arguments are untyped, and its implementation is a common source of bugs).

4.5 Block device drivers

A block device is a device that transfers data in fixed-size blocks and supports random access. Its interface is the block layer's request queue, and its driver implements the blk_mq_ops structure (the multi-queue block layer's operations).

The block layer's design has evolved significantly. The classical block layer (the single-queue block layer) had a single request queue per device, and the requests were processed by a single request_fn function. The single queue was a bottleneck on high-performance devices: the queue's lock serialized the requests, and the single function could not use the device's parallelism. The modern block layer (the multi-queue block layer, introduced in Linux 3.13) has a two-level structure:

• The software queues. One queue per processor core (or per application's submission context). The queue is lockless (or uses a per-queue lock), so the submissions from different cores do not contend.
• The hardware queues. One queue per device's hardware queue (the device's DMA queue, the NVMe's queue pair, the SCSI controller's queue). The hardware queue is what the driver's queue_rq method consumes.

The multi-queue block layer's structure is the reason modern storage devices achieve their throughput: the submissions from different cores go to different software queues, and the software queues are mapped to the hardware queues (typically one per core, if the device has enough queues), so the submissions and the completions do not contend. The mapping is configurable (the nr_requests, scheduler, and rq_affinity parameters in /sys/block/<dev>/queue/), and the administrator can tune it for the workload.

The block driver's queue_rq method is the driver's core: it receives a request (a struct request that describes the operation, the blocks, and the buffer's scatter- gather list), translates it into the device's command format, and submits it to the device. The driver's interrupt handler (or its polling function) detects the completion and calls the block layer's completion function, which processes the request and wakes the waiting process. The block layer's role is to merge the adjacent requests, to schedule them (the I/O scheduler, Lesson 11.5), and to track their completion; the driver's role is to translate them into the device's operations.

4.6 Network device drivers

A network device is a device that transfers data as packets and is addressed by network address rather than by block or byte offset. Its interface is the network stack's packet interface, and its driver implements the net_device_ops structure.

The network driver's interface differs from the character and block drivers' interfaces in three ways:

• The unit of transfer is a packet, not a stream or a block. The driver receives a packet to transmit (through the ndo_start_xmit method) and delivers a received packet to the kernel (by calling the network stack's netif_rx or napi_gro_receive functions). The packet's format is defined by the network protocol (Ethernet, IP, TCP), and the driver's role is to move the packet's bytes between the device and the kernel's socket buffers.
• The interface is not a file. A network device does not have a device node in /dev; its interface is the socket API, and the kernel's network stack is the layer above the driver. An application does not open a network device; it creates a socket, and the kernel's network stack routes the socket's packets to the appropriate device.
• The receive path is asynchronous. The driver does not know when a packet will arrive; the device's interrupt (or the driver's polling, in the NAPI scheme; Lesson 11.2) delivers the packet to the driver, and the driver passes it to the network stack. The transmit path is also asynchronous: the driver's ndo_start_xmit method queues the packet, and the device's completion (an interrupt or a completion descriptor) frees the packet's buffer.

The network driver's structure is shaped by these differences. The driver maintains a ring of receive buffers (the receive descriptors), which the device's DMA engine fills with the received packets; the driver's interrupt handler (or its NAPI polling function) processes the descriptors and passes the packets to the network stack. The driver also maintains a ring of transmit descriptors, which the driver fills with the packets to transmit; the device's DMA engine reads the descriptors and transmits the packets, and the driver's completion handler (an interrupt or a transmit-completion descriptor) frees the packets' buffers. The rings' design is the driver's most important performance decision, and the modern adapters' multiple queues (one per processor core, or one per traffic class) are the source of their throughput.

4.7 The Linux device model

The Linux device model is the kernel's framework for representing the system's devices and their relationships. Its purpose is to let the kernel match devices to drivers, to expose the devices' structure to user space, and to manage the devices' power and their hot-plug events. The model's abstractions are the bus, the device, the driver, and the class:

Bus

A bus is a type of connection that devices can be attached to: the PCI bus, the USB bus, the SCSI bus, the platform bus (for the devices that are built into the system on a chip and that have no discoverable bus). Each bus type defines how its devices are identified (the PCI vendor and device IDs, the USB class codes, the device tree's compatible strings), how they are enumerated (the PCI configuration space, the USB descriptors, the ACPI tables), and how they are matched to drivers.

Device

A device is a physical or virtual component that is attached to a bus. Its representation (the struct device) records its parent, its bus, its driver, its power state, and its properties. The device's structure is the framework's central object: the model's other abstractions refer to it, and its registration is what makes the device visible to the kernel's other subsystems.

Driver

A driver is a piece of code that can operate a device. Its representation (the struct device_driver) records the driver's name, its bus, its probe and remove functions, and its table of supported devices (the IDs that the driver can handle). The driver's registration adds it to the bus's list, and the bus's matching logic connects it to the devices whose IDs it supports.

Class

A class is a type of device that provides a particular interface to user space: the block class (the block devices), the net class (the network interfaces), the tty class (the terminals and serial ports), the input class (the keyboards, mice, and other input devices). A class groups the devices that share an interface, regardless of their bus, and it is the basis for the /sys/class/ hierarchy and for the udev rules that create the device nodes.

The model's sysfs filesystem is its user-space representation. The /sys/ directory mirrors the model's structure: /sys/bus/ contains the bus types, /sys/devices/ contains the devices' hierarchy, /sys/class/ contains the device classes, and /sys/module/ contains the loaded modules. Each device and driver has a directory with its attributes: the device's name, its driver, its power state, and its device-specific attributes (the values that the driver exposes for the administrator's inspection and control). The sysfs files are the administrator's interface to the device model: a read of /sys/class/net/eth0/speed returns the interface's speed, a write to /sys/block/sda/queue/scheduler changes the device's I/O scheduler, and a read of /sys/bus/pci/devices/0000:00:1f.2/driver returns the name of the device's driver.

The model's matching logic is its central mechanism. When a device is registered, the bus's match function compares the device's ID with each of the bus's drivers' ID tables. When a match is found, the bus calls the driver's probe function, which initializes the device. When a driver is registered, the same logic runs in reverse: the bus compares the driver's ID table with each of the bus's devices, and calls the driver's probe for each match. The matching is what makes the system self-configuring: the user plugs in a device, the bus enumerates it, the model matches it to a driver, and the driver initializes it. The mechanism is the reason a modern Linux system can support thousands of devices without the administrator's intervention.

4.8 Device tree and ACPI

The device model needs to know which devices exist before it can match them to drivers. On a discoverable bus (PCI, USB, SCSI), the hardware itself reports the devices' presence and their IDs, so the enumeration is the bus's job. On a non-discoverable bus (the platform bus, the buses on many embedded systems), the hardware does not report the devices' presence, and the system's firmware must describe them. The two firmware descriptions are the device tree and ACPI.

The device tree is a data structure (a tree of nodes and properties) that describes the system's hardware. It is used on the ARM, PowerPC, and RISC-V systems (and on some x86 systems), and it is the standard for the embedded systems and the systems-on-chip. The tree's nodes describe the devices: the CPUs, the memory, the buses, and the devices attached to them. Each node has a compatible property (a list of strings that identify the device's type and its specific model), and the kernel's drivers declare the compatible strings they support. The matching between the two is what connects the devices to their drivers, and it is the device tree's counterpart to the PCI ID table.

ACPI (Advanced Configuration and Power Interface) is the x86 systems' firmware interface. Its tables describe the system's devices, their resources (the memory and I/O regions they use, the interrupts they generate), and their power management. The kernel reads the tables during its initialization and uses them to enumerate the platform devices and to configure their resources. The ACPI tables' _HID and _CID properties are the counterparts of the device tree's compatible strings, and the kernel's drivers declare the ACPI IDs they support. The ACPI's power management interface (the _PS0, _PS3, and _DSM methods) is what the kernel uses to suspend and resume the devices, and it is the basis for the system's power management.

The two mechanisms serve the same purpose on different platforms, and their design reflects their platforms' histories. The device tree is a static description (the tree is built at boot time and does not change), so it is suitable for the embedded systems whose hardware is fixed. ACPI is a dynamic interface (the firmware provides methods that the kernel can call to query and control the devices), so it is suitable for the general-purpose systems whose hardware can change (a laptop with a docking station, a server with hot-plug devices). The kernel abstracts the difference with the fwnode (firmware node) interface, so a driver can be written once and used on both platforms.

4.9 The driver's practical concerns

A driver's code must obey the kernel's rules, and the rules are stricter than a user-space program's. The three most important are the locking rules, the sleeping rules, and the error-handling rules.

Locking. A driver's code can run concurrently on several processor cores, and its interrupt handlers can run concurrently with its process-context code. The shared data structures must therefore be protected. The kernel provides several primitives:

• Spinlocks protect the short critical sections that the interrupt handlers share with the process-context code. A spinlock busy- waits, so it must be held for a very short time (a few instructions to a few dozen), and the code that holds it must not sleep.
• Mutexes protect the longer critical sections that only the process-context code accesses. A mutex sleeps when it cannot be acquired, so the code that holds it may sleep and may block; the code must not hold a mutex in an interrupt handler.
• Atomic operations protect the single variables that the driver shares (a counter, a flag) without the overhead of a lock.
• Per-CPU data avoids the locks entirely for the data that each processor core accesses independently (the statistics, the queue indices).

The locking rules are the driver's most common source of bugs. A driver that accesses a shared structure without a lock is a driver with a race condition; a driver that holds a spinlock for too long is a driver with high latency; a driver that sleeps while holding a spinlock is a driver that can deadlock the system. The kernel's debugging tools (the CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_MUTEXES, and CONFIG_PROVE_LOCKING options, and the lockdep validator) are designed to catch the bugs before they reach production, and the driver's developer should enable them.

Sleeping. The driver's code runs in one of two contexts: the process context (the code that runs on behalf of a system call or a kernel thread) and the interrupt context (the code that runs in an interrupt handler or a softirq; Lesson 11.2). The process context can sleep (it can block, wait for a condition, allocate memory with GFP_KERNEL); the interrupt context cannot (there is no process to wake, and the scheduler's state cannot be saved). The driver's code must therefore be written with its context in mind: the code that runs in the interrupt context must use only the non-blocking operations (the GFP_ATOMIC memory allocation, the spin_lock primitives, the udelay function), and the code that needs to sleep must be deferred to the process context (a workqueue, a kernel thread).

Error handling. The driver's code must handle the device's errors, and the errors are common: a transfer that times out, a device that reports an error, a device that disappears. The driver's error handling must be robust (the device's failure must not crash the kernel), it must be complete (the resources that the driver allocated for the failed operation must be released), and it must be informative (the error must be reported to the administrator through the kernel's log, with enough detail to diagnose the problem). The kernel's dev_err, dev_warn, and dev_info macros are the standard reporting interfaces, and their use is expected of every driver.

4.10 User-space drivers and the alternatives

Not every driver must be in the kernel. The kernel's rules (the locking, the sleeping, the error handling) are demanding, and a buggy driver can crash the system. Several frameworks move the driver's code to user space, where a bug crashes only the driver's process and the development is easier. The frameworks differ in what they move and what they leave in the kernel:

FUSE (Filesystem in Userspace)

A framework that lets a user-space program implement a file system. The kernel's FUSE driver forwards the VFS's operations (the lookups, the reads, the writes) to the user-space program, which performs the operations and returns the results. The framework is used for the file systems that are easier to implement in user space (the network file systems, the archival file systems, the research prototypes), and its performance is lower than a kernel file system's because every operation requires a kernel-to-user-space round trip. The framework's advantage is its safety and its simplicity: a bug in the user-space program crashes the program, not the kernel, and the program can be written in any language.

UIO (Userspace I/O)

A framework that lets a user-space program drive a device. The kernel's UIO driver performs the device's minimal setup (the resource allocation, the interrupt registration) and exposes the device's registers and its interrupts to the user-space program. The program performs the device's operations (the register writes, the DMA setup), and the kernel's role is limited to the setup and the interrupt delivery. The framework is used for the devices whose drivers are easier to write in user space (the industrial I/O devices, the custom hardware), and its performance is close to a kernel driver's because the device's operations do not require a round trip (the program accesses the registers directly).

VFIO (Virtual Function I/O)

A framework that lets a user-space program drive a device with the IOMMU's protection (Lesson 11.3). The kernel's VFIO driver performs the device's minimal setup and the IOMMU mapping, and the user-space program drives the device with the IOMMU's isolation ensuring that the program cannot access the memory it has not been granted. The framework is used for the virtualization's device passthrough (a virtual machine's driver runs in the virtual machine's user space) and for the high-performance user-space drivers (the DPDK and SPDK frameworks). Its performance is close to a kernel driver's, and its safety is higher: the IOMMU prevents the program from corrupting the kernel's memory.

DPDK and SPDK (Data Plane and Storage Performance Development Kits)

Frameworks that let a user-space program drive a high-performance device (a network adapter, a storage device) with a poll-mode design. The frameworks' drivers run entirely in user space, and they poll the device's queues rather than waiting for interrupts, so the per-packet and per-request overhead is minimized. The frameworks are used for the applications that need the highest throughput and the lowest latency (the network functions, the storage servers), and their performance can exceed the kernel's because they avoid the kernel's syscall and interrupt overheads. Their cost is the dedicated processor cores (the poll-mode drivers consume the cores they run on) and the loss of the kernel's abstractions (the frameworks' applications cannot use the kernel's network stack or file systems without a transition).

The frameworks' existence is a statement about the kernel's design: the kernel's abstractions are general-purpose, and their generality costs performance. The frameworks trade the generality for the performance, and they are the right choice for the applications that need it. The kernel's own drivers remain the right choice for the devices that must be shared among the applications, for the devices that the system's other components depend on (the storage devices that hold the file systems, the network adapters that the kernel's stack uses), and for the devices whose drivers are simple enough that the kernel's rules are not a burden.

4.11 Common misconceptions

05 Worked example

The following example traces a character device's read and a block device's request through their drivers, shows the structure of a minimal character driver, and compares the driver's interface with the kernel's abstractions.

#include <linux/module.h>
#include <linux/fs.h>
#include <linux/uaccess.h>

#define DEVICE_NAME "minidev"
#define BUF_SIZE    256

static int    major_number;
static char   device_buffer[BUF_SIZE];
static size_t device_buffer_len;

static int minidev_open(struct inode *inode, struct file *file)
{
    pr_info("minidev: open\n");
    return 0;
}

static ssize_t minidev_read(struct file *file, char __user *buf,
                            size_t count, loff_t *offset)
{
    size_t available;

    if (*offset >= device_buffer_len)
        return 0;                       /* end of data */

    available = device_buffer_len - *offset;
    if (count > available)
        count = available;

    if (copy_to_user(buf, device_buffer + *offset, count))
        return -EFAULT;

    *offset += count;
    return count;
}

static ssize_t minidev_write(struct file *file, const char __user *buf,
                             size_t count, loff_t *offset)
{
    if (count > BUF_SIZE)
        count = BUF_SIZE;

    if (copy_from_user(device_buffer, buf, count))
        return -EFAULT;

    device_buffer_len = count;
    return count;
}

static int minidev_release(struct inode *inode, struct file *file)
{
    pr_info("minidev: release\n");
    return 0;
}

static const struct file_operations minidev_fops = {
    .owner   = THIS_MODULE,
    .open    = minidev_open,
    .read    = minidev_read,
    .write   = minidev_write,
    .release = minidev_release,
};

static int __init minidev_init(void)
{
    major_number = register_chrdev(0, DEVICE_NAME, &minidev_fops);
    if (major_number < 0) {
        pr_err("minidev: cannot register the device\n");
        return major_number;
    }
    pr_info("minidev: registered with major %d\n", major_number);
    return 0;
}

static void __exit minidev_exit(void)
{
    unregister_chrdev(major_number, DEVICE_NAME);
    pr_info("minidev: unregistered\n");
}

module_init(minidev_init);
module_exit(minidev_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("COMP 3150");
MODULE_DESCRIPTION("A minimal character driver");

# List the loaded modules
lsmod | head -20

# Show a module's information
modinfo e1000e 2>/dev/null | head -20

# The output shows the module's filename, its license,
# its author, its description, and its supported devices
# (the PCI IDs, the USB IDs, or the device tree's
# compatible strings).

# Show the devices' drivers in sysfs
ls /sys/bus/pci/drivers/ | head -20
ls /sys/bus/pci/devices/ | head -10

# Show a device's driver
cat /sys/bus/pci/devices/0000:00:1f.2/driver/../uevent 2>/dev/null | head -5
readlink /sys/bus/pci/devices/0000:00:1f.2/driver

# Show the device classes
ls /sys/class/
ls /sys/class/net/
ls /sys/class/block/

# Show the kernel's driver-related messages
dmesg | grep -iE 'driver|module' | head -20

06 Interactive activity

07 Knowledge check

08 Unit discussion

09 Practice exercise

10 Summary

• A device driver is a piece of kernel code that knows how to operate a specific device and that presents the device to the rest of the kernel through one of the kernel's standardized interfaces. The driver is the kernel's translation layer: it receives the kernel's generic requests and turns them into the device's specific operations.
• A loadable kernel module is a piece of kernel code that can be loaded and unloaded at runtime. The module mechanism gives the kernel its flexibility (a device can be supported without a rebuild), its memory efficiency (only the needed modules are loaded), and its development speed (a driver can be tested as a module).
• A driver has four parts: its initialization and cleanup (the module's init and exit functions), its probe and remove (the per-device setup and teardown), its interface implementation (the class's methods), and its interrupt and deferred-work handlers (the top half and the bottom half).
• A character device transfers data as a stream of bytes; its driver implements the file_operations structure (the open, read, write, ioctl, release, and mmap methods). A block device transfers data in fixed-size blocks; its driver implements the blk_mq_ops structure and the multi-queue block layer's request queue. A network device transfers data as packets; its driver implements the net_device_ops structure and the network stack's packet interface.
• The Linux device model represents the system's devices and their relationships. Its abstractions are the bus (the type of connection), the device (the component), the driver (the code that operates it), and the class (the interface that the device provides to user space). Its matching logic connects the devices to the drivers automatically, and its sysfs filesystem exposes the model's structure to user space.
• The device tree and ACPI are the firmware descriptions that the model needs on the platforms whose buses are not discoverable. The device tree is the ARM, PowerPC, and RISC-V systems' static description; ACPI is the x86 systems' dynamic interface. The kernel abstracts the difference with the fwnode interface.
• A driver's code must obey three rules. The locking rules protect the shared data structures from the concurrent access of the several processor cores and the interrupt handlers. The sleeping rules distinguish the process context (which can sleep) from the interrupt context (which cannot). The error-handling rules require the driver to handle the devices' failures robustly, completely, and informatively.
• The user-space driver frameworks (FUSE, UIO, VFIO, DPDK, SPDK) move the driver's code out of the kernel. The frameworks trade the kernel's abstractions for the safety (a bug crashes the application, not the kernel) and, in the poll-mode cases, the performance (the applications avoid the kernel's syscall and interrupt overheads).

11 References

• Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 12, Sections 12.5 and 12.6.
• Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapter 36, I/O Devices. Available online at ostep.org.
• Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 5, Sections 5.2–5.4.
• Corbet, J., Rubini, A., & Kroah-Hartman, G. Linux Device Drivers, 3rd ed. O'Reilly, 2005. The definitive guide to Linux driver development.
• Love, R. Linux Kernel Development, 3rd ed., Chapters 14 and 16.
• Bovet, D. P., & Cesati, M. Understanding the Linux Kernel, 3rd ed. O'Reilly, 2005. Chapter 13, I/O Architecture and Device Drivers.
• Kerrisk, M. The Linux Programming Interface. No Starch Press, 2010. Chapters 14 and 15.
• Linux man-pages: man 2 ioctl, man 4 mem, man 4 null, man 4 zero, man 4 random, man 4 tty, man 4 sd, man 5 sysfs, man 8 insmod, man 8 rmmod, man 8 modprobe, man 8 lsmod, man 8 modinfo, man 1 udevadm, man 8 lspci.