RAID

01 Introduction

Lesson 11.5 presented the disk scheduler and the single device's performance: the seek, the rotational delay, the transfer, and the scheduling algorithms that order the requests. This lesson presents the technique that combines several devices into one logical unit: RAID. The acronym stands for Redundant Array of Inexpensive Disks (or, in the modern usage, Independent Disks), and it was introduced by Patterson, Gibson, and Katz in 1988 in a paper that proposed using several cheap disks in place of one expensive one. The paper's insight was that the several disks could be used not only to improve the storage system's performance (by using them in parallel) but also to improve its reliability (by adding redundancy so that the system survives a disk's failure).

RAID's purpose is therefore twofold, and the two purposes pull in different directions. On the one hand, the several disks can be used in parallel to increase the throughput (the striping technique: the data is spread across the disks, and the requests are served concurrently). On the other hand, the several disks can be used to store redundant information so that the system survives a disk's failure (the mirroring technique: the data is stored on two or more disks, and the system can read from any of them; or the parity technique: an additional piece of data is computed and stored so that a missing piece can be reconstructed). The two techniques can be combined, and the different combinations are the RAID levels.

The RAID levels are the different balances that the designers have chosen. RAID 0 uses the striping alone: it maximizes the performance but provides no redundancy. RAID 1 uses the mirroring: it provides the strongest redundancy at the cost of the storage capacity. RAID 5 uses the striping plus a single parity: it provides the redundancy with a smaller capacity cost, at the price of a more complex write path. RAID 6 uses the striping plus two parities: it survives two disks' failures at the price of a still more complex write path. RAID 10 combines the mirroring and the striping: it provides the strong redundancy and the strong performance at the cost of half the storage capacity.

This lesson presents RAID. It defines the RAID's motivations — the reliability, the availability, and the redundancy — and the storage system's failure model. It presents the three techniques — the striping, the mirroring, and the parity — and the RAID levels that combine them. It presents the RAID levels' performance analysis: the read and write throughput, the small-write problem, and the write-hole. It presents the hardware and software implementations (the RAID controllers, the mdadm, the LVM's RAID targets, the Btrfs's and ZFS's integrated RAID). It presents the rebuild process and its vulnerabilities (the rebuild time, the second failure's risk, the RAID 5 write hole). It presents the RAID's interaction with the solid-state drives (the different failure modes, the write amplification), and it closes with the RAID's design trade-offs and the modern storage systems' choices.

02 Learning objectives

This lesson contributes to U11.6 and U11.7, and to course learning outcomes CLO1, CLO5, and CLO7.

03 Required reading

• Operating System Concepts — Chapter 11, Mass-Storage Structure, Section 11.8, RAID Structure. The standard treatment of the RAID levels, their performance, and their reliability.
• OSTEP — Chapter 38, Redundant Arrays of Inexpensive Disks (RAIDs). OSTEP's treatment of the RAID levels, their evaluation, and their trade-offs is the clearest available account, and it is the basis for the worked example in this lesson.
• Tanenbaum & Bos — Modern Operating Systems, Chapter 5, Section 5.4.3, RAID. A complementary treatment that develops the RAID levels from first principles and compares them systematically.
• Patterson, D. A., Gibson, G., & Katz, R. H. — A Case for Redundant Arrays of Inexpensive Disks (RAID). ACM SIGMOD Conference, 1988. The paper that introduced the RAID terminology and the levels 1 through 5.
• Chen, P. M., Lee, E. K., Gibson, G. A., Katz, R. H., & Patterson, D. A. — RAID: High-Performance, Reliable Secondary Storage. ACM Computing Surveys, 1994. The comprehensive survey of the RAID levels, their performance, and their reliability.
• Leventhal, A. — Triple-Parity RAID and Reed-Solomon Coding. The extensions of the parity technique beyond the RAID 6's two parities (the erasure codes).
• Linux man pages: man 8 mdadm, man 4 md, man 7 md, man 8 lvm, man 8 pvcreate, man 8 vgcreate, man 8 lvcreate, man 5 mdadm.conf, man 8 mkfs.ext4, man 8 mkfs.xfs, man 8 mkfs.btrfs, man 8 zpool (if ZFS is installed), man 1 dmesg, man 8 smartctl, man 1 lsblk.

04 Core content

4.1 Reliability, availability, and redundancy

The RAID's motivation is the storage system's failure and the user's need to survive it. The three concepts are related but distinct:

Reliability

The probability that the storage system delivers the correct data over a specified period, without losing or corrupting it. A reliable system's data is preserved across the failures of its components, and the system's design must account for the failure modes — the disks' mechanical failures, the controllers' electronics failures, the cables' disconnections, and the power supply's outages. The reliability's measure is the mean time to data loss (MTTDL), the expected time between the failures that lose the data.

Availability

The fraction of the time that the storage system is able to serve the requests. An available system continues to serve the requests even when a component fails, either by using the redundant components or by degrading gracefully. The availability's measure is the uptime percentage: a system with 99.9% availability is unavailable for about 8.76 hours per year, and a system with 99.999% availability is unavailable for about 5.26 minutes per year. The RAID levels improve the availability by allowing the system to continue serving the requests while a failed disk is being repaired.

Redundancy

The storage system's technique for surviving a component's failure: the additional data (the mirror's copy or the parity's reconstruction information) that the system stores so that it can reconstruct a failed component's data. The redundancy is the mechanism, and the reliability and the availability are the properties that it provides. The redundancy's measure is the redundancy factor: the ratio of the total storage capacity to the usable capacity (a 2-disk mirror has a redundancy factor of 2; a 5-disk RAID 5 has a redundancy factor of 5/4 = 1.25).

The redundancy's design has three questions: what granularity (the disk's failure, the block's failure, or the byte's failure), what technique (the mirroring, the parity, or the erasure coding), and what layout (the data's and the redundancy's placement across the disks). The RAID levels are the different answers, and their differences are the lesson's subject.

4.2 The three RAID techniques

RAID combines three techniques, each with its own characteristics:

Striping

The data is divided into stripes (small units, typically 4 KB to 1 MB), and the stripes are distributed across the disks in a round-robin order. A logical block's physical location is determined by its logical address: block i is on disk i mod N (for N disks), at the appropriate offset. The striping's purpose is the parallelism: the requests that target different disks are served concurrently, so the system's throughput is the sum of the disks' throughputs. The striping's disadvantage is the reliability: a single disk's failure destroys the entire array's data (the failed disk's stripes are lost, and the array has no way to reconstruct them). The striping's granularity is a design parameter: a fine-grained stripe (a few KB) distributes each request across several disks, so a large request's throughput is high but a small request's latency is low; a coarse-grained stripe (a few hundred KB) keeps each request on one disk, so the small requests' latency is low but the large requests' parallelism is reduced.

Mirroring

The data is stored on two or more disks, and the copies are identical. A write updates all the copies; a read can be served by any of the copies (typically the one whose disk is closest, or the one that is least busy). The mirroring's purpose is the reliability: a single disk's failure leaves the other copies intact, and the system can continue serving the requests from the surviving copies. The mirroring's disadvantage is the capacity cost: a 2-disk mirror uses half the raw capacity for the redundancy, so the usable capacity is half the raw capacity. The mirroring's read performance is the sum of the disks' read throughputs (the reads can be distributed across the copies); its write performance is the minimum of the disks' write throughputs (a write must update all the copies, so the write is as slow as the slowest disk). The mirroring's n-way variant (RAID 1 with three or more copies) provides a stronger reliability at a still higher capacity cost.

Parity

The data is striped across the disks, and an additional parity block is computed and stored for each stripe. The parity is the exclusive-or (XOR) of the stripe's data blocks: P = D₁ ⊕ D₂ ⊕ … ⊕ D_N−1. The parity's purpose is the reliability with a smaller capacity cost than the mirroring's: a single parity block can reconstruct a single failed data block, so the array survives a single disk's failure. The reconstruction is straightforward: the failed block D_i = P ⊕ D₁ ⊕ … ⊕ D_i−1 ⊕ D_i+1 ⊕ … ⊕ D_N−1. The parity's cost is the write amplification: a single data block's update requires the parity block's update as well, and the parity's update requires reading the old data and the old parity (the read-modify-write) or reading the stripe's other data blocks (the reconstruct-write). The parity's design is the source of the RAID 5's small-write problem, which the lesson addresses in section 4.4.

4.3 The RAID levels

The RAID levels combine the three techniques, and each level's design is a different balance. The classical levels (0 through 6) are:

RAID 0 (striping, no redundancy)

The data is striped across N disks with no redundancy. The usable capacity is the sum of the disks' capacities, the read and the write throughputs are N times a single disk's, and the reliability is worse than a single disk's: any disk's failure destroys the entire array's data. RAID 0 is used for the workloads where the performance matters and the data is either replaceable (the scratch space, the build caches) or protected by another mechanism (the application-level replication, the periodic backups).

RAID 1 (mirroring)

The data is stored on two (or more) disks, and the copies are identical. The usable capacity is half the raw capacity (for the 2-disk case), the read throughput is the sum of the disks' (the reads can be distributed), the write throughput is a single disk's (the writes must update both copies), and the reliability is strong: the array survives a single disk's failure. RAID 1 is used for the workloads where the reliability matters and the capacity cost is acceptable (the boot volumes, the database's redo logs, the small high-value datasets).

RAID 2 (bit-level striping with the Hamming code)

A historical level that stripes the data at the bit level and uses the Hamming code for the error correction. RAID 2 is not used in practice (the bit-level striping requires the disks to be synchronized, and the modern disks have their own error correction), and it is mentioned only for completeness.

RAID 3 (byte-level striping with a dedicated parity)

A historical level that stripes the data at the byte level and dedicates a single disk to the parity. RAID 3's design assumes that the requests are large (so the byte-level striping's parallelism is useful) and that the disks are synchronized (so the parity's computation is straightforward). The modern workloads' requests are usually small and the modern disks are not synchronized, so RAID 3 is not used in practice.

RAID 4 (block-level striping with a dedicated parity)

The data is striped at the block level, and a single disk is dedicated to the parity. A read can be served by the data disk that holds the requested block, and a large read can use the parallelism across the data disks. A write requires the parity's update, and the parity disk is the bottleneck: every write to any data disk requires a write to the parity disk, so the parity disk's throughput limits the array's write throughput. The observation is the reason RAID 4 was replaced by RAID 5, which distributes the parity across the disks.

RAID 5 (block-level striping with a distributed parity)

The data is striped at the block level, and the parity is distributed across the disks: for each stripe, one of the disks holds the parity, and the parity's disk rotates from stripe to stripe. The distribution eliminates the RAID 4's parity bottleneck: the writes' parity updates are spread across the disks, so the array's write throughput is higher. The usable capacity is the raw capacity minus one disk's (the parity's overhead: N−1 disks' worth for N disks). The read throughput is N−1 times a single disk's (the data's disks), and the write throughput is reduced by the parity's updates. The reliability is strong for a single disk's failure, and it is the reason RAID 5 is the most common RAID level for the general-purpose storage. The write path's complexity (the read-modify-write or the reconstruct-write) is RAID 5's principal disadvantage, and the RAID 5 write hole (section 4.4) is its most subtle failure mode.

RAID 6 (block-level striping with two parities)

The data is striped at the block level, and two parities are computed and stored for each stripe. The second parity (typically the Reed-Solomon code or the P+Q scheme) allows the array to survive two disks' failures, which is important for the large arrays (the rebuild time for a large array is long, and the second failure's risk is significant during the rebuild). The usable capacity is the raw capacity minus two disks', the write throughput is lower than the RAID 5's (the two parities' updates), and the reliability is the strongest of the classical levels. RAID 6 is used for the large arrays (the storage servers, the backup systems) where the double failure's risk matters.

RAID 10 (a stripe of mirrors)

The disks are grouped into the mirrored pairs (the RAID 1's sets), and the pairs are striped (the RAID 0's arrangement). A write updates both disks of the target pair, and the pairs are used in parallel. The usable capacity is half the raw capacity, the read throughput is the sum of the pairs' (the reads can be distributed across the pairs and within each pair), the write throughput is the sum of the pairs' (the writes to different pairs are concurrent), and the reliability is strong: the array survives a single disk's failure, and (in the common configurations) it can survive several failures as long as no two failures hit the same pair. RAID 10 is used for the workloads where both the performance and the reliability matter (the databases, the transaction-processing systems), and its disadvantage is the capacity cost (half the raw capacity).

The levels' comparison is summarized in the following table. The table's "usable capacity" column assumes N disks of equal size, and the "fault tolerance" column gives the number of the disks' failures that the array survives.

4.4 The small-write problem and the write hole

The RAID 5's parity is the source of its most important performance problem: the small-write problem. When an application updates a single data block in a RAID 5 array, the array must update the corresponding parity block as well, and the parity's update requires the old data and the old parity (or the stripe's other data blocks). There are two strategies:

• Read-modify-write (RMW). The array reads the old data block and the old parity block, computes the new parity (the old parity XOR the old data XOR the new data), and writes the new data and the new parity. The strategy requires two reads and two writes — four I/O operations for a single logical write. The strategy is efficient when the block's disk is idle, because the reads can be issued in parallel with the writes.
• Reconstruct-write (RCW). The array reads the stripe's other data blocks, computes the new parity (the XOR of the other data blocks and the new data), and writes the new data and the new parity. The strategy requires N−1 reads and two writes — one read for each of the stripe's other data blocks. The strategy is efficient when the stripe's other blocks are in the cache (or when the write's data is enough to justify the reads), and it is the preferred strategy for the large writes (the writes that span the whole stripe).

The two strategies' cost is the small-write problem: a single logical write requires several physical operations, and the array's write throughput is reduced. The problem is the reason the RAID 5's write throughput is lower than its read throughput, and it is the reason the RAID 5 is not the right choice for the write-heavy workloads (the databases' transaction logs, the write-intensive applications).

The write hole is a subtler problem. Consider a RAID 5 array with the battery-backed write cache (or the disk's own cache) and a power failure in the middle of a stripe's update. The array's data and its parity are inconsistent: the data block is the new value, but the parity block is the old value (or vice versa), and the array's reconstruction of a failed disk's block would be wrong. The problem is the write hole: the window during which the data and the parity are inconsistent, and the window is closed only when the update completes. The write-hole is the reason the RAID 5's reliability is not perfect: a disk's failure during the window (a power failure followed by a disk's failure, or a crash during the rebuild) can corrupt the array's data.

The write-hole's solutions are:

• The battery-backed write cache. The RAID controller's cache has a battery (or a capacitor), so the cache's contents survive a power failure and the controller can complete the pending updates when the power is restored. The battery-backed cache is the classical solution, and it is the reason the hardware RAID controllers include the battery.
• The journal (the write-intent log). The array writes the update's intent to a journal (a small log on a separate device or on the array itself) before it applies the update, so a crash's recovery can replay the journal and complete the update. The journal is the software RAID's solution, and it is the reason mdadm's RAID 5 has the --write-journal option.
• The copy-on-write (the Btrfs's and ZFS's approach). The file system never overwrites a block in place; every modification writes a new block and updates the tree's root pointer atomically. The copy-on-write avoids the write-hole entirely, because the old data and the old parity remain consistent until the new tree is committed. The approach is the modern storage systems' preferred solution, and it is the reason the Btrfs's and ZFS's RAID implementations are more robust than the classical RAID's.

4.5 The RAID's performance analysis

The RAID levels' performance depends on the workload's access pattern. The following analysis assumes N disks with the same performance characteristics, and it gives the read and write throughputs in the units of a single disk's throughput:

The table's pattern shows the levels' trade-offs. The RAID 0 is the fastest but the least reliable; the RAID 1 and the RAID 10 are the strongest for the reads but the weakest for the writes (the writes must update all the copies); the RAID 5 and the RAID 6 are the balanced choices for the general-purpose workloads, and their weakness is the random writes' parity's overhead.

The analysis is idealized; the real systems' performance depends on the workload's access pattern, the stripe's size, the cache's size, and the controller's design. The key insight is that the RAID levels' performance is not a single number but a function of the workload's characteristics: a level that is fast for the sequential reads may be slow for the random writes, and the system's design must match the level to the workload.

4.6 Hardware and software RAID

The RAID can be implemented in hardware or in software, and the two implementations have different characteristics:

Hardware RAID

A dedicated controller (a RAID card) performs the RAID's operations: it presents the array to the operating system as a single logical device, and it performs the striping, the mirroring, and the parity's computation in its own hardware. The hardware RAID's advantages are the operating system's simplicity (the OS sees a single device, with no RAID's awareness), the controller's capabilities (the battery-backed cache, the dedicated parity's computation, the hot-swap support), and the performance (the controller's hardware offloads the parity's computation from the host's processor). The hardware RAID's disadvantages are the cost (the controllers are expensive), the vendor lock-in (a failed controller must be replaced with the same model, or the array's metadata must be recovered), and the obsolescence (an old controller's firmware may not be supported by the new operating systems).

Software RAID

The host's processor performs the RAID's operations. The Linux kernel's md (multiple device) driver is the classical implementation, and mdadm is its management tool. The software RAID's advantages are the cost (no additional hardware), the portability (the array's metadata is stored on the member disks, so the array can be moved between the systems), and the flexibility (the array's parameters can be changed at runtime, and the modern kernels support the online resizing and the reshaping). The software RAID's disadvantages are the host's processor's overhead (the parity's computation consumes the processor's cycles, which can be significant on the write-heavy workloads) and the host's memory's overhead (the RAID's metadata and its cache consume the memory). The modern processors' performance has reduced the overhead's significance: a modern multi-core processor can perform the RAID 5's parity's computation at the storage device's full throughput, and the software RAID's performance is close to the hardware RAID's for the general-purpose workloads.

The modern storage systems provide additional implementations:

• LVM (Logical Volume Manager). The LVM's RAID targets (the raid0, raid1, raid5, raid6, and raid10 targets) provide the RAID's functionality within the LVM's framework. The LVM's RAID is a software implementation that uses the kernel's dm-raid target, and its advantage is the integration with the LVM's other features (the snapshots, the thin provisioning, the online resizing).
• Btrfs's integrated RAID. The Btrfs file system provides the RAID's functionality as a first-class feature: the administrator specifies the RAID's profile (the single, the raid0, the raid1, the raid10, the raid5, or the raid6), and the file system's allocation policy places the data and the metadata accordingly. The Btrfs's RAID is the copy-on-write's implementation: the data's placement is part of the file system's metadata, and the checksums' verification is the RAID's integrity mechanism. The Btrfs's RAID is more robust than the classical RAID's (the write-hole is avoided by the copy-on-write), and its disadvantage is its maturity (the RAID 5/6 support has had the stability problems).
• ZFS's integrated RAID. The ZFS file system provides the RAID's functionality through its storage pools: the administrator creates a pool from the devices, and the pool's redundancy is configured with the mirrors or the RAID-Z (the RAID 5-like and the RAID 6-like schemes). The ZFS's RAID is the copy-on-write's implementation, with the checksums and the self-healing: the file system detects the corruption (the checksums) and repairs it from the redundant copy (the mirror or the RAID-Z's parity). The ZFS's RAID is the modern storage systems' most thorough implementation, and its disadvantage is its memory's requirements and its licensing (the CDDL is incompatible with the GPL).
• The cloud storage's erasure coding. The distributed storage systems (the Ceph, the Swift, the HDFS) use the erasure coding rather than the classical RAID: the data is divided into the fragments, and the additional fragments (the parity-like codes) are computed and distributed across the storage nodes. The erasure coding's advantage is the flexibility: the administrator can choose the number of the data's fragments (k) and the total fragments (n), and the scheme's fault tolerance is n−k fragments' failures. The erasure coding is the cloud storage's standard because it provides the strong reliability at a small capacity cost, and its implementation's complexity is hidden in the storage system's software.

4.7 The RAID's rebuild and its vulnerabilities

When a disk in a RAID array fails, the array must be rebuilt: the failed disk is replaced, and the array reconstructs the failed disk's data from the surviving disks' data and parity (the RAID 5 and the RAID 6) or from the mirror (the RAID 1 and the RAID 10). The rebuild is a critical period for the array's reliability:

• The rebuild's duration. The rebuild reads the surviving disks' data and writes the reconstructed data to the replacement disk. The rebuild's duration depends on the array's size and the disks' speed: a 4 TB array at 100 MB/s takes about 11 hours, and a 20 TB array takes over 2 days. During the rebuild, the array is vulnerable: the disks are under the heavy load (the rebuild's reads and writes), and the remaining disks' failure would destroy the array's data.
• The second failure's risk. The mean time to failure (MTTF) of the modern disks is typically 1 to 5 million hours, but the annualized failure rate (AFR) is higher, because the disks' failures are not uniformly distributed: the infant mortality and the wear-out effects make the failures cluster. The bit error rate (the probability that a read returns the wrong data without detecting the error) is typically 10⁻¹⁴ to 10⁻¹⁵, and it is significant for the large arrays: a 12 TB disk's full read is 10¹³ bits, so the expected number of the undetected errors is about 0.1 to 1 per full read. The error rate's significance is the reason the large arrays use the RAID 6 or the RAID 10 rather than the RAID 5: the second failure's risk during the long rebuild is real.
• The rebuild's interference. The rebuild's reads and writes compete with the applications' I/O, so the array's performance degrades during the rebuild. The degradation is significant for the RAID 5 and the RAID 6 (the rebuild's reads are the whole array's), and it is smaller for the RAID 1 and the RAID 10 (the rebuild's reads are the surviving mirror's).
• The silent data corruption. A disk can return the wrong data without reporting an error (the silent data corruption, or the bit rot), and the RAID's parity's reconstruction cannot detect it: if the disk returns the wrong data, the array's reconstruction uses the wrong data, and the corrupted block is propagated to the replacement disk. The silent corruption's defence is the checksums (Lesson 10.6): the file system's or the array's checksums detect the corruption, and (in the RAID 6, the RAID 10, or the RAID-Z) the surviving copies allow the correction. The checksums' necessity is the reason the modern storage systems (ZFS, Btrfs) integrate them with the RAID.

The rebuild's vulnerabilities are the reason the modern storage systems' designs have changed:

• The RAID 6 and the erasure coding. The RAID 6's two parities allow the array to survive a second disk's failure during the rebuild, and the erasure coding's higher redundancy (n−k fragments) allows the larger arrays to survive more failures.
• The larger arrays' RAID 10. The RAID 10's mirroring allows the array to survive a disk's failure in each mirror, and the rebuild's duration is shorter (the rebuild reads only the surviving mirror, not the whole array).
• The checksums and the self-healing. The ZFS's and Btrfs's checksums detect the silent corruption, and their redundant copies allow the correction. The self-healing is the modern storage systems' most important reliability feature, and it is the reason the modern systems' RAID is more robust than the classical RAID.
• The proactive maintenance. The modern systems' monitoring (the SMART's attributes, the error counters) allows the administrator to replace a disk before it fails, so the array's rebuild is planned and the failure's risk is reduced. The proactive maintenance is the operational defence, and it complements the technical defences.

4.8 RAID and solid-state drives

The solid-state drives' characteristics change the RAID's design in several ways:

• The failure modes. The SSDs' failures are different from the hard disks': the SSDs tend to fail suddenly (the controller's failure, the firmware's bug, the flash's wear-out), rather than the gradual degradation that the hard disks exhibit. The sudden failures make the proactive maintenance less effective, and they make the array's redundancy more important.
• The write amplification. The SSDs' flash translation layer performs the read-modify-erase-write cycles, and the RAID's parity's updates add to the amplification: the RAID 5's small write requires the additional reads and writes, and the SSD's internal garbage collection's work is increased. The observation is the reason the SSD-based arrays prefer the RAID 1, the RAID 10, and the RAID-Z (the mirroring's designs) over the RAID 5: the mirroring's writes do not require the parity's read-modify-write, so the amplification is smaller.
• The rebuild's speed. The SSDs' rebuild is faster than the hard disks' (the SSDs' random reads are nearly as fast as their sequential reads), so the rebuild's window is shorter and the second failure's risk is smaller. The observation is the reason the SSD arrays can use the RAID 5 more safely than the hard disk arrays.
• The TRIM's propagation. The modern SSDs use the TRIM command to inform the drive which blocks are free, so the drive's garbage collection can reclaim them. The RAID array's TRIM's propagation is a design challenge: the array's logical blocks are the stripes, and the TRIM's information must be translated to the member disks' blocks. The modern Linux kernels support the RAID's TRIM's propagation (the mdadm's --discard option), but the support's performance and correctness vary.

The SSDs' characteristics' implications are the reason the modern storage systems' designs differ for the SSD and the hard disk arrays:

• The SSD arrays often use the mirroring (the RAID 1, the RAID 10, or the RAID-Z's equivalents) rather than the parity, because the mirroring's writes are simpler and the amplification is smaller.
• The SSD arrays use the higher-level integrity (the checksums, the end-to-end protection) to detect the silent corruption, and they rely on the redundant copies to correct it.
• The SSD arrays' controllers (the NVMe devices' internal parallelism, the RAID controllers' SSD-awareness) perform the device-level scheduling and the wear leveling, and the array's design must cooperate with them rather than fight them.

4.9 Common misconceptions

05 Worked example

The following example traces a RAID 5 array's read and write, computes the throughput and the capacity for a concrete configuration, and shows the rebuild's behaviour and its effect on the array's reliability.

Rebuild time = 12 TB ÷ (3 disks × 150 MB/s)
             = 12 × 10^12 ÷ (450 × 10^6) s
             ≈ 26,667 s
             ≈ 7.4 hours

Probability of a second failure in 7.4 hours
  = (7.4 hours ÷ (24 × 365 hours/year)) × AFR
  = (7.4 ÷ 8,760) × AFR
  ≈ 8.4 × 10⁻⁴ × AFR

# Show the array's state
cat /proc/mdstat

# The output shows the array's level (raid5), its
# members (sda1, sdb1, sdc1, sdd1), its state
# ([UUUU] for all members up, [_UUU] for one member
# down), and its rebuild's progress (if any).

# Show the array's details
sudo mdadm --detail /dev/md0

# The output shows the array's level, its chunk size,
# its layout, its members, and its state.

# Show the array's members' SMART status
for disk in sda sdb sdc sdd; do
    echo "=== /dev/$disk ==="
    sudo smartctl -H /dev/$disk 2>/dev/null | tail -2
done

# Show the rebuild's progress
watch -n 5 cat /proc/mdstat

# The rebuild's progress is shown in the mdstat's output
# as the "recovery" or "resync" percentage.

06 Interactive activity

07 Knowledge check

08 Unit discussion

09 Practice exercise

10 Summary

• RAID (Redundant Array of Inexpensive/Independent Disks) combines several devices into one logical unit to improve the storage system's performance and its reliability. Its motivation is the storage system's failure and the user's need to survive it, and the three concepts — the reliability (the data's preservation), the availability (the system's ability to serve the requests), and the redundancy (the mechanism that provides both) — are the design's goals.
• The three RAID techniques are the striping (the data's division into the stripes and their distribution across the disks, for the parallelism), the mirroring (the data's storage on two or more disks, for the reliability), and the parity (the XOR of the stripe's data blocks, for the reliability with a smaller capacity cost). The RAID levels are the different combinations.
• The RAID levels are the RAID 0 (the striping alone: the performance without the redundancy), the RAID 1 (the mirroring: the reliability with the half capacity), the RAID 5 (the striping plus a single distributed parity: the balance of the performance, the capacity, and the reliability), the RAID 6 (the striping plus two parities: the double failure's tolerance), and the RAID 10 (the stripe of mirrors: the performance and the reliability with the half capacity). The historical levels (the RAID 2, the RAID 3, the RAID 4) are not used in practice.
• The small-write problem is the RAID 5's (and the RAID 6's) performance problem: a single data block's update requires the parity's update, and the parity's update requires the read-modify-write or the reconstruct-write. The write hole is the window during which the data and the parity are inconsistent, and a crash during the window can corrupt the array. The write-hole's solutions are the battery-backed write cache, the journal, and the copy-on-write.
• The RAID can be implemented in hardware (the dedicated controller, with the battery-backed cache and the dedicated parity's computation) or in software (the host's processor, with the md driver, the LVM's RAID targets, the Btrfs's and ZFS's integrated RAID). The two implementations have different trade-offs, and the modern processors' performance has reduced the software RAID's overhead's significance.
• The rebuild is the array's reconstruction of a failed disk's data. The rebuild's vulnerabilities are its duration (the long window during which the array has no fault tolerance), the second failure's risk, the silent data corruption's probability, and the rebuild's interference with the applications' I/O. The modern systems' responses are the RAID 6, the erasure coding, the RAID 10, the checksums and the self-healing, and the proactive maintenance.
• The solid-state drives change the RAID's design: the SSDs' sudden failures, the write amplification's significance, the faster rebuilds, and the TRIM's propagation. The SSD arrays often use the mirroring (the RAID 1, the RAID 10, the RAID-Z) rather than the parity, because the mirroring's writes are simpler and the amplification is smaller.
• The modern storage systems have moved beyond the classical RAID: the erasure coding (the n fragments with k data fragments) provides the stronger reliability at a smaller capacity cost, the copy-on-write avoids the write-hole, and the checksums' integration with the RAID provides the end-to-end integrity and the self-healing. The ZFS's and Btrfs's integrated RAID are the modern systems' most thorough implementations.

11 References

• Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 11, Section 11.8.
• Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapter 38, Redundant Arrays of Inexpensive Disks (RAIDs). Available online at ostep.org.
• Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 5, Section 5.4.3.
• Patterson, D. A., Gibson, G., & Katz, R. H. A Case for Redundant Arrays of Inexpensive Disks (RAID). ACM SIGMOD Conference, 1988.
• Chen, P. M., Lee, E. K., Gibson, G. A., Katz, R. H., & Patterson, D. A. RAID: High-Performance, Reliable Secondary Storage. ACM Computing Surveys, 1994.
• Rodeh, O., Bacik, J., & Mason, C. BTRFS: The Linux B-Tree Filesystem. ACM Transactions on Storage, 2013.
• Bonwick, J., & Moore, B. ZFS: The Last Word in File Systems. Sun Microsystems, 2007.
• Linux man-pages: man 8 mdadm, man 4 md, man 7 md, man 5 mdadm.conf, man 8 lvm, man 8 pvcreate, man 8 vgcreate, man 8 lvcreate, man 8 mkfs.ext4, man 8 mkfs.xfs, man 8 mkfs.btrfs, man 8 smartctl, man 1 lsblk.