SSD Architecture

01 Introduction

Lesson 11.1 presented the hard disk's geometry — the platters, the heads, the tracks, and the cylinders — and showed that a read's cost is dominated by the seek and the rotational delay. Lesson 11.5 presented the disk scheduling algorithms that minimize the head's movement, and Lesson 11.6 presented the RAID arrays that combine several disks for the performance and the reliability. This lesson presents the solid-state drive: the device that has no heads to seek and no platters to rotate, and whose characteristics have reshaped the storage stack's design.

An SSD stores data in flash memory, a kind of non-volatile semiconductor memory. The flash's physics differ from the magnetic disks' in ways that matter to the entire stack:

• No mechanical delay. An SSD has no moving parts, so its access time is dominated by the flash memory's read latency (tens of microseconds) and the controller's overhead, and the difference between the sequential and the random access is far smaller than a hard disk's. The observation is the reason the disk scheduling algorithms' seek-reduction (Lesson 11.5) is less important on an SSD.
• Erase-before-write. Flash memory cannot be overwritten in place: a block must be erased before it can be written, and the erase operation works on a much larger unit (the erase block, typically 256 KB to several MB) than the read/write unit (the page, typically 4 KB to 16 KB). The asymmetry is the source of the flash translation layer (FTL), the garbage collection, and the write amplification that shape the SSD's behaviour.
• Limited write endurance. Each flash cell can be erased and rewritten only a limited number of times (typically a few thousand cycles for the consumer drives, more for the enterprise drives). The limit is the source of the wear leveling, the technique that spreads the writes across the cells so that the drive's lifetime is determined by the total number of writes rather than by the hot spots' exhaustion.
• Internal parallelism. An SSD contains many flash chips, and the controller can access several of them in parallel. The parallelism is the source of the SSD's high throughput, and it is the reason the drive can satisfy many concurrent requests without degrading. The NVMe protocol's queue-based interface (Lesson 11.4) is designed to exploit the parallelism.

The four characteristics explain the modern storage stack's design. The flash-specific file systems (Lesson 10.7's F2FS) are designed around the erase-before-write constraint and the internal parallelism. The disk schedulers' adaptation (Lesson 11.5's Kyber and the no-op scheduler) reflects the seek-reduction's reduced importance and the queue-depth's increased importance. The RAID's adaptation (Lesson 11.6's preference for the mirroring over the parity on the SSDs) reflects the write amplification's significance. And the modern file systems' and applications' I/O patterns (sequential writes, aligned writes, the TRIM command) are designed to cooperate with the FTL and the garbage collection rather than to fight them.

This lesson presents the SSD's architecture. It presents the flash memory's fundamentals — the floating-gate cell, the NAND and the NOR technologies, and the SLC, MLC, TLC, and QLC variants. It presents the memory's organization — the cells, the pages, the erase blocks, the planes, the dies, and the channels — and the read, write, and erase operations' asymmetries. It presents the flash translation layer and its address mapping (the page-level, the block-level, and the hybrid), the garbage collection, the write amplification, and the wear leveling. It presents the TRIM command and the drive's internal parallelism. It presents the host interface (the SATA/AHCI and the NVMe), the SSD's performance characteristics (the IOPS, the throughput, the latency, and the tail latency), its reliability (the endurance, the wear-out, and the SMART attributes), and the modern developments (the open-channel SSDs, the ZNS, and the host-managed flash). It closes with the SSD's design trade-offs and the storage stack's future.

02 Learning objectives

This lesson contributes to U11.7 and the unit's synthesis, and to course learning outcomes CLO1, CLO5, CLO6, and CLO7.

03 Required reading

• Operating System Concepts — Chapter 11, Mass-Storage Structure, Section 11.1, Overview of Mass-Storage Structure (the discussion of the solid-state drives), and Section 11.2, Disk Scheduling (the discussion of the flash-based devices' scheduling).
• OSTEP — Chapter 44, Flash-based SSDs. OSTEP's treatment of the flash memory's organization, the flash translation layer, the garbage collection, and the write amplification is the clearest available account, and it is the basis for the worked example in this lesson.
• Tanenbaum & Bos — Modern Operating Systems, Chapter 5, Section 5.4.4, Solid-State Drives. A complementary treatment that develops the flash memory's characteristics from first principles.
• Agrawal, N., Prabhakaran, V., Wobber, T., Davis, J. D., Manasse, M., & Panigrahy, R. — Design Tradeoffs for SSD Performance. USENIX Annual Technical Conference, 2008. The paper that systematically analyzed the SSD's performance characteristics and their trade-offs.
• Gupta, A., Kim, Y., & Urgaonkar, B. — DFTL: A Flash Translation Layer Employing Demand-Based Selective Caching of Page-Level Address Mappings. EuroSys, 2009. The paper that introduced the demand-based FTL mapping.
• Lee, C., Sim, D., Hwang, J., & Cho, S. — F2FS: A New File System for Flash Storage. USENIX FAST, 2015. The design of the flash-specific file system.
• Bjørling, M., Gonzalez, J., & Bonnet, P. — LightNVM: The Linux Open-Channel SSD Subsystem. USENIX FAST, 2017. The design of the open-channel SSD and the host-managed flash.
• Linux man pages: man 8 blkdiscard, man 8 fstrim, man 8 smartctl, man 8 nvme, man 1 fio, man 1 iostat, man 5 proc, man 4 sd, man 8 hdparm, man 1 lsblk, man 8 mkfs.f2fs.

04 Core content

4.1 Flash memory fundamentals

Flash memory is a kind of non-volatile semiconductor memory: it retains its contents when the power is removed, and it can be read, written, and erased electrically. Its storage element is the floating-gate transistor (or, in the modern technologies, the charge-trap cell): a transistor with an additional gate — the floating gate — that is insulated from the rest of the transistor. The floating gate can hold a charge, and the charge's presence or absence determines the transistor's threshold voltage, which the controller's read operation interprets as the cell's value.

The write operation (the program operation) places the charge on the floating gate by applying a high voltage across the transistor's oxide layer; the electrons tunnel through the oxide and are trapped on the gate. The erase operation removes the charge by applying the reverse voltage; the electrons tunnel back through the oxide. The two operations stress the oxide layer, and the layer's degradation is the source of the cell's endurance limit: after a few thousand to a few hundred thousand the program/erase cycles, the oxide is damaged enough that the cell can no longer reliably hold its charge.

The flash memory has two main technologies:

NOR flash

A flash technology that provides the random access (any byte can be read individually), the fast reads, and the slow writes and erases. The NOR flash is used for the code storage (the firmware, the boot loaders, the embedded systems) because the code can be executed in place (the XIP, the execute-in-place). Its capacity is small (the megabytes), its cost per bit is high, and it is not used for the bulk storage.

NAND flash

A flash technology that provides the block access (the reads and the writes are performed on the pages, not the individual bytes), the high density, and the low cost per bit. The NAND flash is used for the bulk storage (the SSDs, the USB drives, the memory cards, the smartphones) because its density and its cost are the best of the flash technologies. Its disadvantages are the block access (the reads and the writes must be the page-aligned), the slow random access (the reads and the writes have a higher latency than the NOR flash's), and the write's and the erase's asymmetry (the write's granularity is the page, and the erase's granularity is the block). The NAND flash is the modern SSDs' technology, and the rest of this lesson refers to it.

4.2 The cell's variants: SLC, MLC, TLC, and QLC

A flash cell's value is determined by the amount of the charge on its floating gate, and the amount can be quantized into the different numbers of the levels. The number of the levels determines the number of the bits the cell stores, and the trade-offs between the density, the performance, and the endurance:

The table's pattern shows the trade-offs. The SLC stores one bit per cell, and its two levels (the charged and the uncharged) are the easiest to distinguish, so the cell's endurance and its performance are the highest. The MLC, the TLC, and the QLC store more bits per cell (the four, the eight, and the sixteen levels), and the levels are the closer together, so the controller's read operation must distinguish them more precisely, and the cell's endurance and its performance are the lower. The higher densities' advantages are the capacity and the cost: a QLC cell stores four times the data of an SLC cell, so the drive's cost per bit is much lower. The consumer drives typically use the TLC and the QLC; the enterprise drives use the MLC and the TLC (and, for the write-intensive workloads, the SLC and the MLC).

The cell's variants' differences explain the modern drives' designs. A drive often uses a hybrid approach: a portion of the cells is configured as the SLC (the fast cache, the write buffer), and the rest as the TLC or the QLC (the bulk storage). The drive's controller writes the incoming data to the SLC cache first (the fast writes), and then migrates the data to the TLC or the QLC storage in the background (the slower writes). The design gives the drive the SLC's performance for the bursty writes and the TLC's or the QLC's capacity for the bulk storage, and it is the reason a modern TLC drive's write throughput can exceed the TLC's native rate for the short bursts.

4.3 The flash memory's organization

A flash chip's memory is organized as a hierarchy of the units:

Cell

The storage element: the floating-gate transistor that holds one or more bits. The cell is the smallest unit, and its state is determined by the charge on its floating gate.

Page

The read and the write unit: a group of the cells (typically 4 KB to 16 KB) that is read or written together. The page is the smallest unit that can be read or written, and its size determines the drive's minimum I/O's granularity.

Erase block

The erase unit: a group of the pages (typically 128 to 1,024 pages, or 256 KB to several MB) that is erased together. The erase block is the smallest unit that can be erased, and its size determines the garbage collection's granularity.

Plane

A group of the erase blocks (typically 1,024 to 4,096) that shares the chip's row and column decoders. A chip has two or four planes, and the planes can be accessed in parallel (the multi-plane operations), so the chip's throughput is multiplied by the planes' count.

Die

A single silicon chip. A package (the physical component) can contain several dies (the stacked dies), and each die has its own planes, its own erase blocks, and its own controller's interface.

Channel

The bus (the electrical connection) between the drive's controller and the dies. A drive has several channels (typically 4 to 16), and the dies on different channels can be accessed in parallel. The channels' count and the dies' count per channel determine the drive's internal parallelism, and the parallelism is the source of the drive's high throughput.

The hierarchy's units' sizes are the source of the flash memory's asymmetries. The read and the write's granularity is the page, and the erase's granularity is the erase block, which is 128 to 1,024 times larger. The observation is the source of the erase-before-write's problem: to update a single page, the drive must read the erase block's other pages, erase the block, and write the updated pages and the unchanged pages back. The read-modify-erase-write's cost is the garbage collection's work, and it is the reason the drive's controller maintains the flash translation layer.

4.4 The read, write, and erase operations

The flash memory's three operations have different characteristics:

• The read. The read operation senses the cell's threshold voltage and compares it with the reference levels, so it determines the cell's value. The read's granularity is the page, and its latency is tens of microseconds (the SLC's reads are the fastest, and the QLC's are the slowest). The read's endurance is effectively unlimited (the reads do not stress the oxide layer), though the read disturb effect (the reads' repeated voltage's application can shift the neighbouring cells' thresholds) requires the controller to refresh the data periodically.
• The write (the program). The write operation places the charge on the cell's floating gate, so it sets the cell's value. The write's granularity is the page, and its latency is hundreds of microseconds (the SLC's writes are the fastest, and the QLC's are the slowest). The write's endurance is limited: each write stresses the oxide layer, and the layer's degradation is the source of the endurance limit. The write's constraint is the erase-before-write: the cell must be erased before it can be written, and the erase's granularity is the erase block.
• The erase. The erase operation removes the charge from the cells, so it resets the erase block to its initial state. The erase's granularity is the erase block, and its latency is milliseconds (the slowest of the three operations). The erase's endurance is the lowest: each erase stresses the oxide layer, and the layer's degradation is the source of the endurance limit.

The three operations' asymmetries are the SSD's design's central constraints:

The table's pattern explains the SSD's design. The read's page granularity and its short latency are why the drive's reads are fast. The write's page granularity and its longer latency are why the drive's writes are slower than its reads. The erase's block granularity and its long latency are why the drive cannot update a page in place: the controller must write the new data to a fresh page and invalidate the old one, and the erase block's reclamation requires the garbage collection's read-modify-erase-write cycle.

4.5 The flash translation layer

The flash translation layer (FTL) is the drive's controller's firmware that hides the flash memory's constraints behind the block device's interface. Its principal function is the address translation: it maps the host's logical block addresses (LBAs) to the flash's physical page numbers (PPNs), and it updates the mapping on every write.

The FTL's translation is the source of the drive's out-of-place update behaviour: when the host overwrites a logical block, the FTL does not overwrite the corresponding physical page (the flash memory cannot); instead, it writes the new data to a fresh physical page, updates the mapping to point to the new page, and marks the old page as invalid. The old page's space is not reclaimed until the garbage collection (the GC) runs.

The FTL's mapping has three principal designs:

Page-level mapping

The FTL maintains a mapping entry for every logical page, so the translation is direct and the write's placement is unconstrained (any free page can be used). The page-level mapping's advantage is its flexibility; its disadvantage is the mapping table's size: for a 1 TB drive with 4 KB pages, the table has 256 million entries, and at 4 bytes per entry, the table is 1 GB — a significant fraction of the drive's RAM.

Block-level mapping

The FTL maps the logical blocks (not the pages) to the physical blocks, and the pages' offsets within the blocks are preserved. The mapping table is much smaller (one entry per block rather than per page), but the write's placement is constrained: the logical block's page must be written to the same offset in the physical block, so the FTL must read the old block's other pages, erase the block, and write the updated pages back — the read-modify-erase-write cycle on every small write. The block-level mapping's write amplification is high.

Hybrid mapping

The FTL combines the two: a small portion of the drive (the log blocks, or the cache) uses the page-level mapping, and the rest (the data blocks) uses the block-level mapping. The write's hot spots are served by the log blocks (the fast, flexible writes), and the cold data is stored in the data blocks (the compact mapping). The hybrid's advantage is the balance between the mapping table's size and the write amplification; its disadvantage is the complexity of the log blocks' management (the merge operations).

The modern drives use a variant of the page-level mapping with the demand-based caching: the full mapping table is stored in the flash (the drive's NAND), and only the working set (the recently accessed pages' mappings) is cached in the drive's RAM (the DRAM or the SRAM). The design (the DFTL, the demand-based FTL) reduces the RAM's requirement while keeping the page-level mapping's flexibility, and it is the modern consumer drives' standard.

4.6 Garbage collection and write amplification

The garbage collection (GC) is the FTL's process of reclaiming the invalid pages' space. The GC selects a victim erase block (typically the block with the fewest valid pages, to minimize the copies), reads the block's valid pages, writes them to a new block, erases the victim, and updates the mapping to point to the valid pages' new locations. The GC's work is the source of the SSD's write amplification:

The write amplification factor (WAF) is the ratio of the physical writes to the logical writes. A WAF of 1 means the drive writes exactly the data the host requested; a WAF of 3 means the drive writes three times the data (the host's writes plus the GC's copies plus the metadata's updates). The WAF is the SSD's central performance and lifetime metric: a higher WAF means a lower effective throughput (the drive's bandwidth is consumed by the GC's copies) and a shorter lifetime (the extra writes consume the cells' endurance).

The WAF depends on the workload's access pattern:

• The sequential writes. A sequential write fills the erase blocks completely, so the GC's victim blocks have few valid pages (or none), and the GC's copies are minimal. The sequential writes' WAF is close to 1.
• The random writes. A random write scatters the updates across the erase blocks, so the victim blocks have many valid pages, and the GC must copy them. The random writes' WAF is higher, and it grows with the drive's utilization (a fuller drive has fewer free blocks, so the GC runs more often and copies more data).
• The over-provisioning. The drive's over-provisioning is the extra capacity (beyond the host's usable capacity) that the FTL uses for the GC's workspace. A larger over-provisioning reduces the WAF (the GC has more free blocks to work with), and it is the reason the enterprise drives (with the higher over-provisioning) have the lower WAFs and the longer lifetimes.
• The TRIM command. The TRIM (section 4.8) informs the drive which logical blocks are free, so the FTL can invalidate the corresponding physical pages immediately (rather than waiting for the host to overwrite them). The TRIM reduces the WAF by reducing the GC's valid pages' count.

The WAF's typical values are 1.1 to 2 for the sequential workloads, 2 to 5 for the random workloads, and higher for the write-heavy, high-utilization workloads. The values' significance is the reason the modern file systems and the applications are designed to write sequentially and to issue the TRIM: the designs reduce the WAF, which improves the drive's performance and its lifetime.

4.7 Wear leveling

The wear leveling is the FTL's process of spreading the writes across the flash cells, so that no cell is worn out before the others. Without the wear leveling, a workload that repeatedly writes the same logical blocks would wear out the corresponding physical cells, and the drive would fail long before its total capacity's endurance was consumed. The wear leveling's goal is to make the drive's lifetime depend on the total number of writes rather than on the hot spots' concentration.

The wear leveling has two forms:

• Dynamic wear leveling. The FTL chooses the free blocks for the new writes in a way that spreads the writes across the blocks' erase counts: it prefers the blocks with the lowest erase counts, and it avoids the blocks with the highest counts. The dynamic wear leveling is simple, and it handles the active data (the data that is written frequently), but it does not handle the cold data (the data that is written rarely and never moved): the cold data's blocks are never rewritten, so their erase counts stay low, and the active data's blocks wear out while the cold data's blocks are underused.
• Static wear leveling. The FTL periodically moves the cold data to the blocks with the higher erase counts (or to the fresh blocks), so the cold data's blocks' erase counts increase and the drive's wear is spread more evenly. The static wear leveling's cost is the extra copies (the cold data's migration), and its benefit is the longer lifetime for the workloads with the mixed hot and cold data. The modern drives use the static wear leveling (or its variants), and the technique is the reason the drives' lifetimes are close to the cells' rated endurance.

The wear leveling's effectiveness is measured by the wear leveling factor: the ratio of the drive's maximum erase count to its average. A factor of 1 means the wear is perfectly even; a factor greater than 1 means some cells are worn more than others. The modern drives' factors are close to 1 (typically 1.1 to 1.3), and the SMART's attributes (the Wear_Leveling_Count and the Media_Wearout_Indicator) report the drive's wear's state.

4.8 The TRIM command

The TRIM (also called the discard) is the command by which the host informs the drive which logical blocks are no longer in use. The command's purpose is to reduce the write amplification: without it, the FTL cannot distinguish a logical block that holds the valid data from one that holds the stale data (the file system deleted the file, but the drive does not know), so the GC copies the stale pages as if they were valid. With the TRIM, the FTL can invalidate the corresponding physical pages immediately, so the GC's valid pages' count is smaller and the copies are fewer.

The TRIM's implementation has three forms:

• The ATA TRIM command. The SATA drives' TRIM command (the DATA SET MANAGEMENT command), which the host issues for a range of logical blocks. The command's transport is the ATA protocol, and its granularity is the drive's logical block size.
• The SCSI UNMAP command. The SAS drives' UNMAP command, which the host issues for a list of the logical block ranges. The command is the SCSI's equivalent of the ATA's TRIM.
• The NVMe Dataset Management command. The NVMe drives' command, which the host issues for a list of the logical block ranges (the dataset management, the DSM). The command is the NVMe's equivalent of the ATA's TRIM, and it is more flexible (the host can specify the ranges' attributes, including the deallocate and the write-zeroes).

The Linux host's TRIM's mechanisms are:

• The mount option. The discard mount option makes the file system issue the TRIM for the blocks that it frees, so the drive's FTL is informed immediately. The option's disadvantage is the TRIM's per-operation overhead (the command's issue for each free), which can reduce the performance on the workloads with the many small frees.
• The periodic fstrim. The fstrim command (or the systemd's fstrim.timer) issues the TRIM for all the free blocks periodically (typically the weekly), so the drive's FTL is informed in the batches. The approach's advantage is the lower per-operation overhead; its disadvantage is the TRIM's delay (the blocks are not invalidated until the periodic run).
• The blkdiscard command. The blkdiscard command issues the TRIM for a whole device or a range, so the drive's FTL invalidates the corresponding pages. The command is useful for the devices' preparation (the secure erase, the thin provisioning's reclamation) and for the testing.

The TRIM's importance is the reason the modern file systems and the storage stacks support it: without it, the drive's WAF is higher, its performance is lower, and its lifetime is shorter. The TRIM's propagation through the RAID and the LVM's layers (Lesson 11.6) is a design challenge, and the modern kernels support it (the mdadm's --discard option, the LVM's issue_discards option) for the devices that support it.

4.9 The SSD's internal parallelism

The internal parallelism is the source of the SSD's high throughput. A drive contains many flash chips (the dies), and the controller can access several of them in parallel:

• The channel-level parallelism. The drive's controller has several channels (typically 4 to 16), and the dies on different channels can be accessed concurrently. The channels' count determines the drive's channel-level parallelism.
• The die-level parallelism. The dies on the same channel can be interleaved (the controller issues the operations to the different dies' registers, and the operations execute concurrently), so the channel's throughput is multiplied by the dies' count.
• The plane-level parallelism. The planes within a die can be accessed concurrently (the multi-plane operations), so the die's throughput is multiplied by the planes' count.
• The cell-level parallelism. The drive's controller can access the different pages' offsets concurrently (the plane-level and the block-level interleaving), so the drive's throughput is multiplied by the interleaving's degree.

The parallelism's effect is the reason the drive's throughput depends on the queue depth: a single request can use only one channel (or one die, or one plane), so its throughput is limited; several concurrent requests can use the different channels, so the drive's aggregate throughput is the sum. The observation is the reason the NVMe protocol's queue-based interface (Lesson 11.4) is the modern drives' standard: the host can submit thousands of the requests concurrently, and the drive's controller can schedule them across its channels, so the drive's parallelism is fully exploited. The observation is also the reason the disk schedulers' role (Lesson 11.5) is reduced on the SSDs: the scheduler's job is to keep the queue's depth high, not to reorder the requests for the seek's reduction.

4.10 The host interfaces: SATA/AHCI and NVMe

The SSD's host interface determines the drive's performance ceiling and the host's ability to exploit the drive's parallelism:

SATA / AHCI

The SATA (Serial ATA) interface and its AHCI (Advanced Host Controller Interface) command protocol. The SATA's bandwidth is up to 6 Gb/s (about 600 MB/s after the encoding), and the AHCI's command queue has a single queue with 32 slots. The interface was designed for the hard disks (the AHCI's single queue reflects the hard disk's single actuator), and it is the bottleneck for the modern SSDs: the 600 MB/s bandwidth limits the drive's sequential throughput, and the single queue with the 32 slots limits the drive's parallelism's exploitation.

NVMe

The NVMe (Non-Volatile Memory Express) interface and its protocol. The NVMe runs over the PCIe (Lesson 11.1), so its bandwidth is the PCIe's (tens of gigabytes per second). Its command interface is the queue-based design: the drive has thousands of the command queues (the queue pairs), each with thousands of the command slots, and the host submits the commands by writing them to the queues in the memory (the DMA, Lesson 11.3). The queue-based interface is designed for the flash memory's parallelism: the host can submit the thousands of the requests concurrently, and the drive's controller can schedule them across the channels, so the parallelism is fully exploited. The NVMe is the modern high-performance SSDs' standard, and its interface is one of the reasons the NVMe SSDs achieve the millions of the IOPS with the low latency.

The interface's differences explain the modern drives' designs. A SATA SSD's performance is limited by the SATA's bandwidth and the AHCI's queue's depth, so its throughput is at most about 550 MB/s and its IOPS is limited by the queue's depth. An NVMe SSD's performance is limited only by the flash memory's characteristics, so its throughput can be several gigabytes per second and its IOPS can be millions. The observation is the reason the modern high-performance systems use the NVMe devices and the SATA devices are relegated to the legacy systems and the low-performance applications.

4.11 The SSD's performance characteristics

The SSD's performance is characterized by four metrics:

• The sequential throughput. The drive's bandwidth for the large, sequential reads and writes. The SATA SSDs' sequential throughput is about 500–550 MB/s; the NVMe SSDs' is 2,000–7,000 MB/s (and the PCIe 5.0 devices' is higher). The sequential throughput is limited by the interface's bandwidth (the SATA) or the flash memory's bandwidth (the NVMe), and it is the metric that the large file transfers and the backups care about.
• The random IOPS. The drive's operations per second for the small, random requests. The SATA SSDs' random IOPS is about 50,000–100,000 (limited by the AHCI's queue's depth); the NVMe SSDs' is 500,000–1,500,000 (the higher-end devices' is more). The random IOPS is limited by the flash memory's parallelism and the controller's scheduling, and it is the metric that the databases and the virtualization care about.
• The latency. The time an individual request takes. The SSDs' average latency is tens of microseconds (the SATA's is slightly higher than the NVMe's), and the latency's distribution has the tail (the 99th and the 99.9th percentiles) that is much higher than the average. The tail latency is the source of the modern systems' quality-of- service (QoS) concerns: a system that must bound the worst-case latency must account for the SSD's tail.
• The queue depth's scaling. The drive's throughput as a function of the number of the concurrent requests. The SSD's throughput increases with the queue's depth until it saturates at the drive's maximum (the parallelism's limit). The SATA's queue's depth is limited to 32, so its scaling saturates early; the NVMe's queue's depth is thousands, so its scaling continues further. The observation is the reason the modern systems' applications and the storage stacks are designed to submit the many concurrent requests (the asynchronous I/O, the deep queues), and it is the reason the NVMe's queue-based interface is the modern drives' standard.

The performance characteristics' analysis is the basis for the modern storage stack's design. The file systems' allocation policies (Lesson 10.5) write sequentially and align the writes to the flash memory's pages, so the WAF is reduced and the throughput is improved. The disk schedulers (Lesson 11.5) keep the queue's depth high, so the drive's parallelism is exploited. The applications use the asynchronous I/O (the io_uring, the libaio) and the deep queues, so the drive's parallelism is fully used. And the modern systems' monitoring (the iostat, the fio, the smartctl) tracks the performance's metrics and the drive's health, so the degradation is detected early.

4.12 The SSD's reliability

The SSD's reliability is characterized by the endurance, the failure modes, and the SMART attributes:

• The endurance. The drive's total data that can be written before the cells wear out, expressed as the terabytes written (TBW) or the drive writes per day (DWPD) over the warranty's period. The endurance depends on the cells' variant (the SLC, the MLC, the TLC, the QLC), the drive's over-provisioning, and the workload's WAF. The consumer drives' endurance is typically 0.3 to 1 DWPD; the enterprise drives' is 1 to 10 DWPD. The endurance is the drive's lifetime's limit, and the workload's WAF determines how quickly the limit is reached.
• The failure modes. The SSDs' failures differ from the hard disks': the SSDs tend to fail suddenly (the controller's failure, the firmware's bug, the flash's wear-out), rather than the gradual degradation that the hard disks exhibit. The sudden failures make the proactive maintenance less effective, and they are the reason the SSDs' redundancy (the RAID, the backups) is essential. The SSDs also have the read disturb (the reads' repeated voltage's application can shift the neighbouring cells' thresholds) and the retention (the charge can leak over the years, especially at the high temperatures) as their specific failure modes, and the modern controllers manage both (the periodic refresh, the retention's management).
• The SMART attributes. The SSDs' SMART attributes report the drive's health and its wear:
  • Wear_Leveling_Count (the ID 177): the drive's average erase count (or the remaining life's percentage). The attribute decreases as the drive wears out.
  • Media_Wearout_Indicator (the ID 233): the drive's remaining life's percentage. A value of 100 means the drive is new; a value of 0 means the drive is at its rated endurance's limit.
  • Total_LBAs_Written (the ID 241): the total data that the drive has written, in the logical blocks. The attribute lets the administrator compute the drive's remaining endurance.
  • Reallocated_Sector_Ct (the ID 5): the number of the sectors that the drive has reallocated to the spare area. The attribute's growth is the drive's degradation's signal.
  • Program_Fail_Count and Erase_Fail_Count (the IDs 171 and 172): the number of the program and erase failures. The attributes' growth is the drive's wear-out's signal.

4.13 The modern developments: the open-channel SSDs, the ZNS, and the host-managed flash

The SSD's architecture is evolving, and the modern developments move the flash management from the drive's controller to the host:

Open-channel SSDs

The drives that expose the flash memory's physical interface to the host (the channels, the dies, the blocks, and the pages), rather than the block device's logical interface. The host's software (the host-managed FTL) performs the address translation, the garbage collection, and the wear leveling, so the host can optimize the flash's management for its workload. The open-channel SSDs' advantages are the host's control (the host can align its writes to the flash's pages and blocks, so the WAF is minimized) and the drive's simplicity (the controller's firmware is simpler, so the drive is cheaper and more reliable). The framework's disadvantages are the host's complexity (the host must manage the flash's details) and the ecosystem's immaturity (the host's software must be written for the drives). The Linux kernel's LightNVM subsystem is the open-channel SSDs' support, and the pblk is the kernel's host-managed FTL.

Zoned Namespaces (ZNS)

The NVMe's standard for the zoned storage: the drive's logical address space is divided into the zones, and each zone must be written sequentially (the writes within a zone must start at the zone's beginning and proceed in order). The ZNS's advantage is the alignment with the flash memory's characteristics: the sequential writes reduce the WAF, and the zones' management is simpler than the open-channel SSDs' (the drive still manages the FTL, but the host's writes' pattern is constrained). The ZNS is the modern standard's approach to the host-managed flash, and it is supported by the Linux kernel (the null_blk's ZNS mode, the zonefs file system, and the application's frameworks).

Computational storage

The drives that perform the computation within the storage device (the data's filtering, the compression, the encryption, the search), so the host's processor is offloaded and the data's movement is reduced. The computational storage is an emerging area, and its designs vary from the simple (the fixed-function offloads) to the programmable (the drives with the embedded processors and the software). The approach's promise is the performance (the data's processing at the storage's speed) and the efficiency (the reduced data's movement), and its challenges are the standardization and the software's ecosystem.

4.14 Common misconceptions

05 Worked example

The following example traces a write through the FTL, computes the write amplification for different workloads, estimates the drive's lifetime, and shows the parallelism's effect on the throughput.

Total physical writes = 1.07 TB × 1,000
                      ≈ 1,070 TB
                      ≈ 1.07 PB

Total logical writes = 1,070 TB ÷ 1.5
                     ≈ 713 TB

Total logical writes = 1,070 TB ÷ 4
                     ≈ 268 TB

DWPD = 713 TB ÷ (1 TB × 365 days × 5 years)
     ≈ 0.39

Single request (4 KB read):
  Throughput = 4 KB ÷ 100 µs = 40 MB/s

Single request (4 KB write):
  Throughput = 4 KB ÷ 500 µs = 8 MB/s

64 concurrent requests (4 KB reads):
  Throughput = 64 × 4 KB ÷ 100 µs = 2,560 MB/s

64 concurrent requests (4 KB writes):
  Throughput = 64 × 4 KB ÷ 500 µs = 512 MB/s

# Identify the SSD and its interface
lsblk -d -o NAME,SIZE,ROTA,TRAN,MODEL
# ROTA=0 means non-rotational (an SSD).

# Show the drive's SMART health and its wear
sudo smartctl -a /dev/nvme0n1 2>/dev/null | head -60
sudo smartctl -a /dev/sda 2>/dev/null | grep -iE 'wear|erase|written|reallocat'

# The output's key attributes:
#   Wear_Leveling_Count — the drive's wear
#   Media_Wearout_Indicator — the remaining life's percentage
#   Total_LBAs_Written — the total logical writes
#   Reallocated_Sector_Ct — the reallocated sectors

# Show the drive's queue parameters
cat /sys/block/nvme0n1/queue/nr_requests
cat /sys/block/nvme0n1/queue/scheduler
cat /sys/block/nvme0n1/queue/rotational

# Issue a TRIM for the free blocks
sudo fstrim -v /
sudo fstrim -v /home

# The output shows the number of bytes that were trimmed.

# Measure the drive's performance with fio
sudo fio --name=seqread --filename=/tmp/fio-test.bin \
    --size=1G --rw=read --bs=1M --direct=1 --runtime=10 \
    2>/dev/null | grep -E 'bw|IOPS'

sudo fio --name=randread --filename=/tmp/fio-test.bin \
    --size=1G --rw=randread --bs=4k --iodepth=64 \
    --ioengine=libaio --direct=1 --runtime=10 \
    2>/dev/null | grep -E 'bw|IOPS'

06 Interactive activity

07 Knowledge check

08 Unit discussion

09 Practice exercise

10 Summary

• Flash memory is a kind of non-volatile semiconductor memory. Its storage element is the floating-gate transistor, and its value is determined by the charge on the gate. The two main technologies are the NOR flash (the random access, the fast reads, the code storage) and the NAND flash (the block access, the high density, the bulk storage). The cell's variants — the SLC, the MLC, the TLC, and the QLC — store one, two, three, or four bits per cell, and the trade-offs among them are the density, the performance, and the endurance.
• The flash memory is organized as a hierarchy: the cells, the pages (the read and write unit), the erase blocks (the erase unit), the planes, the dies, and the channels. The read's and the write's granularity is the page, and the erase's granularity is the erase block, which is 128 to 1,024 times larger. The asymmetry is the source of the erase-before-write's constraint and the flash translation layer.
• The flash translation layer (FTL) maps the host's logical block addresses to the flash's physical page numbers, and it performs the out-of-place updates: a write allocates a new physical page and invalidates the old one. The FTL's mapping designs are the page-level (the flexible, the large table), the block-level (the compact, the high write amplification), and the hybrid (the balance). The modern drives use the page-level mapping with the demand-based caching.
• The garbage collection (GC) reclaims the invalid pages' space by copying the victim block's valid pages to a new block, erasing the victim, and updating the mapping. The GC's work is the source of the write amplification (the ratio of the physical writes to the logical writes). The WAF is the SSD's central performance and lifetime metric, and it depends on the workload's access pattern (the sequential writes' WAF is close to 1; the small random writes' is 2 to 4 or higher).
• The wear leveling spreads the writes across the cells. The dynamic wear leveling chooses the free blocks by their erase counts; the static wear leveling periodically moves the cold data to the blocks with the higher erase counts. The TRIM command informs the drive which logical blocks are free, so the FTL can invalidate the corresponding physical pages and the GC's copies are fewer. The internal parallelism (the channels, the dies, the planes) is the source of the SSD's high throughput, and it is the reason the drive's throughput depends on the queue's depth.
• The SSD's host interfaces are the SATA/AHCI (the single queue with the 32 slots, the 600 MB/s bandwidth, the hard disk's legacy) and the NVMe (the thousands of the queue pairs, the PCIe's bandwidth, the flash's parallelism). The NVMe is the modern high-performance SSDs' standard, and it is the reason the NVMe SSDs achieve the millions of the IOPS with the low latency.
• The SSD's performance is characterized by the sequential throughput, the random IOPS, the latency (the average and the tail), and the queue depth's scaling. The SSD's reliability is characterized by the endurance (the TBW and the DWPD), the failure modes (the sudden failures, the read disturb, the retention), and the SMART attributes (the Wear_Leveling_Count, the Media_Wearout_Indicator, the Total_LBAs_Written, the Reallocated_Sector_Ct, the Program_Fail_Count, and the Erase_Fail_Count).
• The modern developments move the flash management from the drive's controller to the host: the open-channel SSDs (the host's software performs the FTL's work), the Zoned Namespaces (the zones with the sequential writes' requirement), and the computational storage (the drives that perform the computation). The developments are the recognition of the semantic gap between the host's knowledge and the drive's: the host knows the data's hotness and the free blocks' state, while the drive's controller must infer the same information from the block-level requests.
• The SSD's characteristics have reshaped the storage stack's design: the file systems write sequentially (to reduce the WAF), the disk schedulers keep the queue's depth high (to exploit the parallelism), the RAID's designs prefer the mirroring over the parity (to reduce the write's cost), and the TRIM's propagation keeps the FTL informed. The storage stack's designs are the response to the SSD's characteristics, and they are the reason the modern systems' performance is so much higher than the hard-disk-based systems'.

11 References

• Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 11, Sections 11.1 and 11.2.
• Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapter 44, Flash-based SSDs. Available online at ostep.org.
• Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 5, Section 5.4.4.
• Agrawal, N., Prabhakaran, V., Wobber, T., Davis, J. D., Manasse, M., & Panigrahy, R. Design Tradeoffs for SSD Performance. USENIX Annual Technical Conference, 2008.
• Gupta, A., Kim, Y., & Urgaonkar, B. DFTL: A Flash Translation Layer Employing Demand-Based Selective Caching of Page-Level Address Mappings. EuroSys, 2009.
• Lee, C., Sim, D., Hwang, J., & Cho, S. F2FS: A New File System for Flash Storage. USENIX FAST, 2015.
• Bjørling, M., Gonzalez, J., & Bonnet, P. LightNVM: The Linux Open-Channel SSD Subsystem. USENIX FAST, 2017.
• Bjørling, M. Zone Append: A New Way of Writing to Zoned Storage. USENIX FAST, 2020.
• Linux man-pages: man 8 blkdiscard, man 8 fstrim, man 8 smartctl, man 8 nvme, man 1 fio, man 1 iostat, man 5 proc, man 4 sd, man 8 hdparm, man 1 lsblk, man 8 mkfs.f2fs.