Direct Memory Access

01 Introduction

Lesson 11.1 presented the I/O subsystem's hardware and its three data-transfer mechanisms. Programmed I/O transfers each byte or word with the processor's own load and store instructions, so the processor is occupied for the entire transfer. Interrupt-driven I/O frees the processor during the device's operation, but the processor still moves the data, and each transfer costs an interrupt. Direct memory access (DMA) removes the processor from the data path entirely: a separate DMA engine transfers the data between the device and memory, and the processor is involved only at the transfer's setup and its completion.

The saving is enormous. A 1 MB transfer at 100 MB/s takes 10 ms. If the processor moved the data with programmed I/O, it would spend those 10 ms executing load and store instructions — millions of cycles that could have been spent on the application's work. With DMA, the processor spends a few hundred cycles setting up the transfer and a few hundred handling its completion, and the remaining cycles are available for other work. The observation is why DMA is used by every high-throughput device: hard disks, solid-state drives, network adapters, graphics cards, and the high-speed buses that connect them.

DMA's design has three parts. The first is the DMA engine — the hardware that performs the transfer. It may be a separate chip (the classical 8237 DMA controller) or, on modern systems, a part of the device's own controller (the bus master DMA engine in an NVMe SSD, a network adapter, or a graphics card). The second is the transfer's description — the addresses, the length, and the direction that the processor writes to the DMA engine (or to a descriptor in memory) to start the transfer. The third is the address translation — the mapping from the buffer's virtual address to the physical addresses that the DMA engine uses, which the operating system handles with contiguous allocation, scatter-gather lists, or an IOMMU.

This lesson presents DMA in detail. It defines DMA and explains why it exists. It presents the DMA controller and the two kinds of DMA engines (the external controller and the bus-mastering device controller). It presents the DMA transfer cycle — setup, transfer, and completion — and the descriptors and descriptor rings that modern devices use to describe their transfers. It presents the physical-address problem and the three solutions: contiguous allocation (the bounce buffer), scatter-gather DMA, and the IOMMU. It presents cache coherency — the interaction between DMA and the processor's caches — and the distinction between coherent and streaming DMA mappings. It presents DMA in the Linux kernel, the NVMe queue pairs that exploit DMA's parallelism, and DMA's role in virtualization. It closes with the DMA's latency and throughput characteristics and the modern devices that push them to their limits.

02 Learning objectives

This lesson contributes to U11.3 and U11.4, and to course learning outcomes CLO1, CLO5, and CLO6.

03 Required reading

• Operating System Concepts — Chapter 12, I/O Systems, Section 12.2.4, Direct Memory Access, and Section 12.7, Performance. The standard treatment of the DMA mechanism, the DMA controller, the transfer cycle, and the performance implications.
• OSTEP — Chapter 36, I/O Devices. OSTEP's treatment of DMA as the third data-transfer mechanism, and its discussion of the DMA engine and the transfer's setup and completion, is the clearest available account, and it is the basis for the worked example in this lesson.
• Tanenbaum & Bos — Modern Operating Systems, Chapter 5, Section 5.1.3, Direct Memory Access. A complementary treatment that develops the DMA controller and the transfer cycle from first principles.
• Corbet, J., Rubini, A., & Kroah-Hartman, G. — Linux Device Drivers, 3rd ed. O'Reilly, 2005. Chapter 15, Memory Mapping and DMA. The Linux kernel's DMA API: the coherent and streaming mappings, the DMA pools, the scatter-gather mappings, and the cache-coherency rules.
• Bovet, D. P., & Cesati, M. — Understanding the Linux Kernel, 3rd ed. O'Reilly, 2005. Chapter 13, I/O Architecture and Device Drivers. The Linux kernel's DMA structures and the bus-mastering device model.
• Love, R. — Linux Kernel Development, 3rd ed., Chapter 14, Block Devices I/O, and Chapter 16, The Block I/O Layer. The kernel's block layer and the request queue that the DMA engine consumes.
• Intel Corporation. — Intel Virtualization Technology for Directed I/O Architecture Specification. The authoritative reference for the IOMMU and its role in DMA translation and device isolation.
• Linux man pages: man 5 proc (for /proc/interrupts and the /sys interfaces), man 1 lspci, man 8 ethtool, man 1 iostat, man 1 perf, man 8 smartctl, man 5 vfio (if available), man 1 nvme.

04 Core content

4.1 What DMA is and why it exists

Direct memory access (DMA) is the mechanism by which a device transfers data to or from memory without the processor's per-byte involvement. The processor sets up the transfer — it writes the memory address, the length, and the direction to the DMA engine — and the engine performs the transfer, reading from or writing to memory directly. When the transfer is complete, the engine raises an interrupt, and the processor handles the completion.

The mechanism's purpose is to free the processor from the data movement. The alternative — programmed I/O — requires the processor to execute a load or a store for every byte or word, and the processor's time is proportional to the amount of data transferred. For a device that transfers megabytes per second, the processor's time is enormous: a 1 GB transfer at 1 GB/s takes one second of device time, but the processor's per-byte involvement would consume hundreds of millions of cycles — most of the processor's capacity for that second. DMA reduces the processor's involvement to a setup and a completion, and the remaining cycles are available for the application's work.

The saving's size depends on the device's rate and the processor's speed:

The table is illustrative — the actual numbers depend on the processor, the bus, and the device — but the pattern is what matters. DMA reduces the processor's involvement by five orders of magnitude for a 1 MB transfer, and the reduction grows with the transfer's size. The observation is why DMA is used by every high-throughput device: the processor's time is too valuable to spend on moving bytes.

4.2 The DMA engine and bus mastering

The DMA engine is the hardware that performs the transfer. Its design has two forms:

External DMA controller (third-party DMA)

A separate chip that sits between the devices and memory. The classical example is the Intel 8237 DMA controller, which was used in the IBM PC and its successors. The processor programs the controller with the transfer's parameters, and the controller performs the transfer by arbitrating with the processor for the memory bus. The controller is shared among the devices, and each device has a DMA channel through which it requests the controller's service. External DMA controllers are still used for the low-bandwidth devices (the legacy floppy controller, the parallel port, some embedded systems), but they have been replaced by the bus-mastering devices for the high-bandwidth ones.

Bus-mastering device controller (first-party DMA)

The device's own controller contains the DMA engine, and the controller is the bus master: it arbitrates with the processor for the memory bus directly, without an external controller's mediation. The classical examples are the SCSI controller, the ATA controller, and the modern NVMe SSD, network adapter, and graphics card. The bus-mastering design has three advantages: it scales with the number of devices (each device has its own engine, so the engines do not contend for a shared controller), it is faster (the device's engine can be optimized for the device's characteristics), and it is more flexible (the engine can implement descriptor-based transfers, scatter-gather, and other features that a shared controller cannot). Modern systems use bus-mastering devices almost exclusively.

The two designs differ in the bus master — the component that controls the bus during the transfer. In third-party DMA, the external controller is the master; in first-party DMA, the device's controller is the master. The distinction matters because the bus master determines the transfer's capabilities: a bus-mastering device can perform descriptor-based and scatter-gather transfers, while an external controller's transfers are typically limited to a single contiguous buffer.

The bus master's role also determines the transfer's relationship to the processor. While the DMA engine is the bus master, the processor cannot use the bus; the engine's accesses and the processor's accesses are interleaved by the bus's arbitration protocol. On modern systems, the interleaving is efficient: the memory controller and the bus's protocol allow the DMA engine and the processor to share the memory's bandwidth with little contention. On older systems, the DMA engine's accesses stole cycles from the processor — the condition known as cycle stealing — and the processor's execution slowed during the transfer. The modern systems' arbitration has largely eliminated the problem, but it is worth understanding: a DMA transfer consumes memory bandwidth, and the bandwidth is a shared resource.

4.3 The DMA transfer cycle

The DMA transfer cycle has three phases: setup, transfer, and completion.

Setup. The processor's driver prepares the transfer:

1. Allocate the buffer. The driver allocates a memory buffer for the transfer (or uses a buffer that the file system or the network stack has already allocated).
2. Map the buffer. The driver translates the buffer's virtual address to the physical addresses that the DMA engine uses. The mapping may be a single contiguous range (if the buffer is physically contiguous) or a list of ranges (a scatter-gather list). The mapping's details are the subject of sections 4.4 and 4.5.
3. Write the descriptor. The driver writes the transfer's description — the source or destination addresses, the length, the direction, and the device's control bits — to the DMA engine's registers or to a descriptor in memory.
4. Start the transfer. The driver writes the engine's "start" bit (or, for a descriptor-based device, the doorbell register), and the engine begins the transfer.

Transfer. The DMA engine performs the transfer. It reads the descriptor (if it is in memory), then reads or writes the data one word at a time, arbitrating with the processor for the memory bus. The engine's accesses are physical: it does not use the processor's MMU, so it cannot fault, cannot be preempted, and cannot follow a page table. The constraint is the source of the physical-address problem.

Completion. When the transfer is complete, the engine updates its status register and raises an interrupt. The driver's handler (the top half, Lesson 11.2) reads the status register, confirms the transfer's success, unmaps the buffer (releasing the physical addresses' reservation), and schedules the deferred work (the bottom half) that processes the completed request.

The three phases' durations differ enormously. The setup and completion take a few hundred to a few thousand cycles each, and the transfer takes as long as the data's movement requires. For a 1 MB transfer at 1 GB/s, the transfer takes about 1 ms (about 3 million cycles at 3 GHz), while the setup and completion together take about 2,000 cycles. The observation is why the DMA's overhead is negligible for large transfers and significant for small ones: a 64-byte transfer's setup and completion may take longer than the transfer itself, and the driver may prefer programmed I/O for the smallest transfers (Lesson 11.1, section 4.4).

4.4 The physical-address problem

The physical-address problem is the mismatch between the addresses that the processor's programs use and the addresses that the DMA engine uses. A program's buffer is specified by a virtual address, which the processor's MMU translates to a physical address through the page table. A DMA engine, however, does not use the processor's MMU: it addresses memory by physical address, and it cannot fault, cannot be preempted, and cannot follow a page table. The engine can therefore transfer only to or from the physical addresses that the driver has programmed, and the driver must supply them.

The translation is complicated by two facts:

• The buffer's virtual pages may be scattered across physical memory. A 64 KB buffer may occupy 16 physically non-contiguous 4 KB pages, because the virtual-memory system does not guarantee that a process's pages are physically contiguous. A DMA engine that expects a single contiguous range cannot transfer the buffer directly.
• The buffer's pages may be moved or evicted during the transfer. The virtual-memory system may page out a page, migrate it for compaction or NUMA balancing, or map a different page in its place. A DMA engine that is transferring to or from the page would then read or write the wrong memory. The driver must therefore pin the buffer's pages — mark them as unmovable and unevictable — for the transfer's duration.

The operating system solves the problem in one of three ways, each with its own cost and its own applicability: contiguous allocation, scatter-gather DMA, and the IOMMU.

4.5 The three solutions

Contiguous allocation (the bounce buffer)

The kernel allocates a physically contiguous buffer for the transfer, copies the data between the buffer and the program's scattered pages, and performs the DMA on the contiguous buffer. For a write (memory to device), the kernel copies the program's data into the bounce buffer before the transfer; for a read (device to memory), the kernel copies the buffer's data into the program's pages after the transfer. The approach works for any device — even one that supports only a single contiguous range — but it costs an extra copy, which is significant for high-bandwidth transfers. For a 1 GB/s transfer, the extra copy consumes memory bandwidth and the processor's time, and it can halve the effective throughput. The approach is used as a fallback: the kernel uses it when the device cannot handle a scatter-gather list, or when the buffer's addresses are outside the range that the device can address (a DMA zone constraint on some 32-bit devices).

Scatter-gather DMA

The device's DMA engine accepts a scatter-gather list (SG list), a list of (physical address, length) pairs that describe the buffer's pages. The engine reads the list and transfers each page separately, so no copy is needed. The approach is the standard for modern storage and network controllers: an NVMe SSD, a SATA controller with AHCI, a SCSI controller, and a modern network adapter all support scatter-gather. The engine's descriptor format varies — the AHCI controller uses a command list with a scatter- gather table, the NVMe controller uses a command with a physical region page (PRP) list or a scatter-gather list, the network adapter uses a ring of buffer descriptors — but the principle is the same: the driver describes the buffer's pages, and the engine transfers them without a copy.

IOMMU (I/O memory management unit)

The IOMMU is a hardware component that translates the device's addresses the way the processor's MMU translates the program's addresses. The kernel programs the IOMMU with a mapping from the device's address space to physical memory, and the device uses virtual addresses (from its own perspective) that the IOMMU translates. The device can then transfer to or from a buffer whose pages are scattered, because the IOMMU performs the translation that the engine cannot. The approach has three advantages: it removes the need for a contiguous buffer (the device sees a contiguous address space even when the physical memory is scattered), it removes the need for a bounce buffer (the device can address any of the system's memory), and it provides isolation (a device can access only the memory that the kernel has mapped for it, so a buggy or malicious device cannot corrupt arbitrary memory). The IOMMU is standard on modern x86 systems (Intel VT-d, AMD Vi) and on ARM systems (the SMMU), and it is essential for virtualization (section 4.9).

The three solutions are not mutually exclusive. A modern system uses the IOMMU where it is available, scatter-gather lists where the device supports them, and bounce buffers as a fallback. The driver's choice depends on the device's capabilities, the buffer's characteristics, and the kernel's configuration.

4.6 Cache coherency and DMA

The DMA engine and the processor both access memory, and both may cache the data they access. The processor's caches (L1, L2, L3) hold copies of the memory's contents, and the DMA engine's accesses may not be visible to them. The result is a cache-coherency problem with two directions:

• The device writes memory, but the processor's cache holds a stale copy. If the processor has recently read the buffer, its cache may hold the buffer's old contents. The DMA engine writes the new contents to memory, but the processor's subsequent read is served from the cache, and it sees the old contents. The processor must invalidate the cache lines before reading the new data.
• The processor writes memory, but the DMA engine reads the old contents. If the processor has recently written the buffer, its write may still be in the cache and not yet in memory. The DMA engine reads the memory and sees the old contents. The processor must flush (write back) the cache lines before the transfer begins.

The two operations are the cache maintenance that the driver must perform, and their necessity depends on the hardware's cache-coherency model:

Hardware-coherent DMA

The hardware maintains the caches' coherency with the DMA engine's accesses. The bus's protocol snoops the processor's caches during the DMA transfer: when the engine writes memory, the protocol invalidates the corresponding cache lines; when the engine reads memory, the protocol flushes the processor's dirty cache lines first. The hardware's coherency means the driver does not need to perform any cache maintenance, and the DMA transfer's correctness is automatic. x86 systems are hardware-coherent for the devices on the PCIe bus, and most modern ARM systems are coherent for the devices on the coherent interconnect (the CCI or CCN). The coherence is a property of the bus and the platform, not of the device.

Software-managed coherency

The hardware does not maintain the caches' coherency, and the driver must perform the cache maintenance explicitly. The driver invalidates the cache lines before reading the device's data and flushes the cache lines before the device reads the processor's data. The software-managed approach is used by some embedded systems, by some ARM systems without a coherent interconnect, and by some devices (typically the older or the simpler ones) on otherwise-coherent systems. The maintenance's cost is the cache operations' overhead, and its correctness requires the driver to get the operations' order and extent right — a subtle and error-prone task.

The Linux kernel's DMA API abstracts the two models. The driver uses the same API regardless of the hardware's coherency, and the kernel's implementation performs the cache maintenance when it is needed:

Coherent DMA mappings

A mapping that the driver creates once and uses repeatedly, with the guarantee that the processor and the device always see the same data. The kernel allocates the buffer (through dma_alloc_coherent) and maps it in a way that is coherent on all platforms: on a hardware-coherent system, the mapping is a simple address translation; on a software-managed system, the kernel allocates the buffer from a region that is not cached (or manages the cache explicitly). Coherent mappings are used for the long-lived structures that the driver and the device share — the descriptor rings, the command queues, the status blocks — where the coherence's guarantee is essential and the buffer's reuse justifies the allocation's cost.

Streaming DMA mappings

A mapping that the driver creates for a single transfer and destroys when the transfer is complete. The driver maps the buffer (through dma_map_single for a single range or dma_map_sg for a scatter-gather list), performs the transfer, and unmaps the buffer (through dma_unmap_single or dma_unmap_sg). The kernel performs the cache maintenance at the map and unmap operations: on a hardware-coherent system, the operations are mostly no-ops; on a software-managed system, the kernel flushes or invalidates the cache lines as needed. Streaming mappings are used for the transient buffers that the driver passes to the device — the file system's pages, the network stack's packet buffers — where the buffer's lifetime is short and the coherence's guarantee is needed only for the transfer's duration.

The distinction between the two mappings is the driver author's most important DMA decision. A coherent mapping's buffer is always safe to access from the processor and the device, but its allocation is expensive and its region may be limited (the coherent pool is a fixed-size region, and its size is a boot-time parameter). A streaming mapping's buffer is cheap to create, but the driver must not access it between the map and the unmap (the access would race with the device), and the driver must perform the mapping and unmapping in the correct order. A driver that uses the wrong kind of mapping — a coherent mapping for a transient buffer, a streaming mapping for a long-lived structure — is a driver with poor performance or subtle bugs.

4.7 DMA descriptors and ring buffers

A modern high-performance device does not perform one transfer at a time; it processes a stream of transfers, and it needs a way to describe them to the DMA engine and to report their completion. The mechanism is the descriptor ring (or queue): a circular array of descriptors in memory, each describing one transfer, with a pair of producer and consumer indices that the driver and the device use to coordinate.

The ring's operation is a classic producer-consumer protocol:

1. The driver writes a descriptor. The driver fills the next free descriptor with the transfer's parameters: the buffer's physical addresses (or the scatter-gather list's address), the length, the operation (read or write), and any device-specific flags. The driver then advances the producer index.
2. The driver notifies the device. The driver writes to the device's doorbell register (or, for NVMe, to the submission queue's tail doorbell), which tells the device that new descriptors are available.
3. The device reads the descriptors. The device's DMA engine reads the descriptors from the ring (using DMA, of course), processes the transfers, and advances the consumer index. It may process several descriptors in one batch, and it may use its own scheduling to order them.
4. The device reports the completion. The device writes a completion entry to a separate ring (the completion queue) and, if the driver has requested it, raises an interrupt. The completion entry identifies the completed transfer and reports its status.
5. The driver processes the completions. The driver's bottom half (Lesson 11.2) reads the completion entries, frees the buffers, and completes the requests. It advances the ring's consumer index (or, for NVMe, the completion queue's head doorbell), which tells the device that the entries have been processed and the slots can be reused.

The ring's design has three important properties. First, it is lock-free (or nearly so): the driver and the device each own one index, and the coordination is through the indices' values rather than through locks. Second, it is batched: the device can process several descriptors in one pass, and the driver can submit several descriptors before notifying the device, so the per-transfer overhead is amortized. Third, it is parallel: a device can have several rings (one per queue, one per processor core), and the driver can submit to different rings from different cores without contention. The parallelism is the source of the modern devices' throughput: an NVMe SSD with 64 queues, each with 1,024 entries, can have 65,536 transfers in flight at once, and the driver can submit them from 64 cores concurrently.

The NVMe protocol's queue pairs are the canonical modern example. Each queue pair consists of a submission queue (the driver writes the commands) and a completion queue (the device writes the completions), and each queue pair is associated with a separate interrupt (an MSI-X vector, Lesson 11.2). The driver can create one queue pair per processor core, submit the I/O requests from the core that originated them, and receive the completions on the same core — the per-core queue design that eliminates the cross-core synchronization that a single queue would require. The design is the reason NVMe SSDs achieve millions of IOPS with low latency, and it is the reason the storage stack has been redesigned around it.

4.8 DMA and virtualization

The IOMMU's translation has a second role beyond the physical-address problem: it provides isolation. A device's DMA engine can access only the memory that the kernel has mapped for it in the IOMMU, so a buggy or malicious device cannot corrupt arbitrary memory. The isolation is the basis for the modern virtualization features:

• Device passthrough. A virtual machine can be given direct access to a physical device (a network adapter, a storage controller), and the IOMMU translates the device's DMA to the virtual machine's memory. The virtual machine's driver talks to the device directly, without the hypervisor's mediation, and the device's performance is near-native. The IOMMU's isolation ensures that the device can access only the virtual machine's memory, not the hypervisor's or another virtual machine's.
• SR-IOV (Single Root I/O Virtualization). A physical device can present several virtual functions, each of which can be assigned to a different virtual machine. Each virtual function has its own DMA engine and its own IOMMU mapping, so the virtual machines' traffic is isolated and the device's performance is shared.
• VFIO (Virtual Function I/O). The Linux framework that exposes a device's IOMMU mapping to user space, so a user-space driver (a virtual machine's emulator, a DPDK application) can program the device's DMA directly. VFIO is the basis for the high-performance user-space I/O frameworks, and it relies on the IOMMU for its safety.

Without the IOMMU, the device passthrough would be unsafe: a device could DMA to any physical address, including the hypervisor's memory, and a malicious virtual machine could use the device to attack the host. The IOMMU's isolation is what makes the modern virtualization's performance possible, and it is one of the reasons the IOMMU is standard on modern systems.

4.9 DMA latency and throughput

DMA's performance characteristics are its latency and its throughput:

• The setup and completion latency. The driver's setup takes a few hundred to a few thousand cycles (the buffer's allocation, the mapping, the descriptor's write, and the doorbell's write). The completion's handling takes a similar time (the interrupt, the status read, the buffer's unmap, and the deferred work's scheduling). For a small transfer, the setup and completion's latency dominates the transfer's cost, and the driver may prefer programmed I/O or interrupt-driven I/O. For a large transfer, the latency is negligible.
• The transfer's throughput. The engine's throughput is limited by the device's rate, the bus's bandwidth, and the memory's bandwidth. The three are usually matched: a PCIe 4.0 ×4 device can transfer about 8 GB/s, the memory's bandwidth is tens of gigabytes per second, and the device's own rate (an NVMe SSD's 7 GB/s, a 100 Gb/s network adapter's 12.5 GB/s) determines which one is the bottleneck. The engine's descriptor processing adds overhead — the engine must read each descriptor, translate its addresses (if an IOMMU is present), and update its status — so the effective throughput is lower than the bus's raw bandwidth. The overhead is small for large descriptors (a 1 MB transfer's descriptor is processed once) and significant for small ones (a 4 KB transfer's descriptor is processed once, and the descriptor's processing may take longer than the transfer).
• The memory bandwidth's contention. The DMA engine's accesses and the processor's accesses share the memory's bandwidth. A high-throughput DMA transfer can consume a significant fraction of the memory's bandwidth, and the processor's accesses may slow as a result. The contention is usually small on modern systems (the memory's bandwidth is much larger than the devices' aggregate rate), but it can become significant on a system with many high-speed devices (a server with several NVMe SSDs and a 100 Gb/s network adapter), and the system's design must account for it.

The characteristics explain the modern stack's design. The large transfers are the DMA's domain, and the drivers use large descriptors, scatter-gather lists, and descriptor rings to amortize the per-transfer overhead. The small transfers are the processor's domain, and the drivers use programmed I/O or interrupt-driven I/O for them. The boundary between the two is a tunable parameter, and the drivers expose it (the network adapter's tx-usecs and rx-usecs coalescing parameters, the block layer's max_sectors_kb) so the administrator can tune it for the workload.

4.10 Common misconceptions

05 Worked example

The following example traces a 1 MB read from an NVMe SSD through the DMA machinery, computes the transfer's cost, compares the three address-mapping solutions, and shows how the descriptor ring's design affects the device's throughput.

# Show the PCI devices' DMA capabilities
lspci -vv 2>/dev/null | grep -E 'DMA|MSI|Capabilities' | head -30

# The output shows whether the device supports bus-mastering
# DMA, scatter-gather, MSI/MSI-X, and 64-bit addressing.

# Show the network adapter's DMA and offload capabilities
sudo ethtool -k eth0 2>/dev/null | grep -E 'scatter|dma|sg'

# Show the block device's queue parameters
cat /sys/block/nvme0n1/queue/max_sectors_kb
cat /sys/block/nvme0n1/queue/max_segments
cat /sys/block/nvme0n1/queue/nr_requests
cat /sys/block/nvme0n1/queue/scheduler

# The max_segments parameter is the maximum number of
# scatter-gather entries the device supports.

# Show the IOMMU's status
dmesg | grep -iE 'iommu|vt-d|dmar' | head -20

# Show the system's DMA-related kernel parameters
cat /proc/cmdline | tr ' ' '\n' | grep -iE 'iommu|dma'

06 Interactive activity

07 Knowledge check

08 Unit discussion

09 Practice exercise

10 Summary

• Direct memory access (DMA) is the mechanism by which a device transfers data to or from memory without the processor's per-byte involvement. The processor sets up the transfer, the DMA engine performs it, and the engine raises an interrupt when it is complete. DMA's purpose is to free the processor from the data movement, and its saving grows with the transfer's size.
• The DMA engine is the hardware that performs the transfer. It may be an external DMA controller (the classical 8237, shared among the devices) or a bus-mastering device controller (the modern NVMe SSD, network adapter, and graphics card, each with its own engine). Modern systems use the bus-mastering design almost exclusively.
• The DMA transfer cycle has three phases: setup (the driver allocates the buffer, maps it, writes the descriptor, and starts the engine), transfer (the engine moves the data), and completion (the engine raises an interrupt, and the driver handles it). The setup and completion take a few hundred to a few thousand cycles; the transfer takes as long as the data's movement requires.
• A descriptor describes one transfer; a descriptor ring (or queue) is a circular array of descriptors with a producer-consumer protocol. The ring is lock-free, batched, and parallel, which makes it suitable for high-performance devices. NVMe's queue pairs are the canonical modern example: each pair has a submission queue and a completion queue, and each is associated with a separate MSI-X interrupt.
• The physical-address problem is the mismatch between the processor's virtual addresses and the DMA engine's physical addresses. The operating system solves it with contiguous allocation (a bounce buffer and an extra copy), scatter-gather DMA (a list of the buffer's pages), or an IOMMU (a hardware translation that removes the need for both). The three solutions are not mutually exclusive: a modern system uses the IOMMU where it is available, scatter-gather lists where the device supports them, and bounce buffers as a fallback.
• Cache coherency is the interaction between the DMA and the processor's caches: the device's writes may not be visible to the processor's cache, and the processor's writes may not be visible to the device. A hardware-coherent system maintains the coherency automatically; a software-managed system requires the driver to perform the cache maintenance explicitly. The Linux kernel's DMA API abstracts the difference with coherent mappings (for long-lived structures) and streaming mappings (for transient buffers).
• The IOMMU provides address translation and isolation. The isolation is the basis for the modern virtualization features: device passthrough (a virtual machine gets direct access to a physical device), SR-IOV (a physical device presents several virtual functions), and VFIO (the Linux framework that exposes a device's IOMMU mapping to user space). Without the IOMMU, the device passthrough would be unsafe.
• DMA's performance characteristics are its setup and completion latency (a few hundred to a few thousand cycles) and its transfer throughput (limited by the device, the bus, and the memory's bandwidth). The per-transfer overhead is amortized over the data's transfer for large transfers and dominates for small ones; the driver's threshold is a tunable parameter.
• The modern stack's design reflects the DMA's characteristics: large descriptors, scatter-gather lists, and descriptor rings for the large transfers; programmed I/O or interrupt-driven I/O for the small ones; the IOMMU for the address translation and the isolation; and the per-core queue pairs for the parallelism. The design is the reason the modern devices achieve their throughput and their low latency.

11 References

• Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 12, Sections 12.2.4 and 12.7.
• Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapter 36, I/O Devices. Available online at ostep.org.
• Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 5, Section 5.1.3.
• Corbet, J., Rubini, A., & Kroah-Hartman, G. Linux Device Drivers, 3rd ed. O'Reilly, 2005. Chapter 15, Memory Mapping and DMA.
• Bovet, D. P., & Cesati, M. Understanding the Linux Kernel, 3rd ed. O'Reilly, 2005. Chapter 13, I/O Architecture and Device Drivers.
• Love, R. Linux Kernel Development, 3rd ed., Chapters 14 and 16.
• Intel Corporation. Intel Virtualization Technology for Directed I/O Architecture Specification. The authoritative reference for the IOMMU.
• Linux man-pages: man 1 lspci, man 8 ethtool, man 5 proc, man 1 iostat, man 1 perf, man 8 smartctl, man 8 nvme, man 1 trace-cmd.