COMP 3150Operating Systems

01 Introduction

Lesson 10.4 presented the directory as a data structure and showed how a name is looked up, added, and removed. Lesson 10.5 presented file allocation and showed how a file's logical blocks are mapped to physical blocks and how the free space is tracked. Both lessons described operations that modify several on-disk structures at once: creating a file updates the inode bitmap, the inode table, the directory entry, and the superblock's free-inode count. This lesson addresses the problem that those multi-structure updates create: what happens if the system crashes in the middle of one?

The crash-consistency problem is the problem of keeping a file system's on-disk metadata consistent when a crash interrupts an update. A file creation is not a single disk write; it is a sequence of writes to different blocks. If the system crashes after the first write but before the last, the file system is left in a state that is neither the old state nor the new one. A directory entry may name an inode that the inode bitmap still marks as free; a block may be claimed by two files; a file may have grown without the free-block count being decremented. The file system's metadata is corrupt, and without a recovery mechanism, the only fix is to run a consistency checker that scans the entire file system and repairs the inconsistencies it finds.

The classical recovery mechanism is fsck (the file system consistency check), which scans the file system's metadata, cross-checks the structures against one another, and repairs the inconsistencies. fsck is effective, but it is slow: it must read the entire file system, and its cost grows with the file system's size. On a large file system, fsck can take hours, and the longer it takes, the longer the system is unavailable after a crash. The journal is the modern mechanism that avoids the full scan: the file system records its intended metadata changes in a dedicated log before it applies them, so after a crash it can replay the log (a few hundred kilobytes to a few megabytes) and reach a consistent state in seconds.

This lesson presents the crash-consistency problem and the classical fsck response. It presents the write-ahead logging protocol, the principle on which journaling is based, and the journal itself — a circular log of transactions. It presents the journaling modes (writeback, ordered, and data), which trade performance against the strength of the consistency guarantee. It presents JBD2, the journaling layer used by ext3 and ext4, and the recovery process that replays the journal after a crash. It presents the modern alternatives — log-structured file systems and copy-on-write file systems such as Btrfs and ZFS — which achieve crash consistency by a different route. It closes with the performance and durability trade-offs that shape a file system designer's choice of mechanism.

Prerequisites

This lesson builds directly on Lessons 10.1 through 10.5. The inode as the file's metadata record, the directory entry as a name-to-inode mapping, and the allocation of a file's data blocks are all assumed. The journal's design assumes the file system's metadata structures — the superblock, the inode table, the bitmaps, and the directories — and the operations that modify them. The discussion of copy-on-write assumes the block-level view of a file from Lesson 10.5. Familiarity with the Linux command line from Lesson 2.2 is helpful, since the interactive activity uses standard Unix tools.

02 Learning objectives

By the end of this lesson you will be able to

  • Explain the crash-consistency problem and why a multi-block metadata update can leave a file system corrupt.
  • Describe the write-ahead logging protocol and the journal's structure as a circular log of transactions.
  • Compare the journaling modes (writeback, ordered, data) and explain the consistency, durability, and performance trade-offs of each.
  • Describe how a journaling file system recovers after a crash by replaying the journal, and explain the role of the commit record and the checkpoint.
  • Explain log-structured file systems and copy-on-write file systems as alternatives to journaling, and compare them with the journaling approach.
  • Explain the role of checksums in detecting metadata corruption and the limitations of journaling in preventing it.

This lesson contributes to U10.6 and U10.7, and to course learning outcomes CLO1, CLO5, and CLO6.

03 Required reading

  • Operating System Concepts — Chapter 14, Section 14.7, Recovery, and Chapter 15, Section 15.6, Log-Structured File Systems. The standard treatment of consistency checking, journaling, and log-structured file systems.
  • OSTEP — Chapter 42, Crash Consistency: FSCK and Journaling. OSTEP's treatment of the crash-consistency problem, the cost of fsck, and the write-ahead logging protocol is the clearest available account, and it is the basis for the worked example in this lesson.
  • Tanenbaum & Bos — Modern Operating Systems, Chapter 4, Section 4.3.5, Journaling File Systems. A complementary treatment that develops the journal's design from first principles.
  • Tweedie, S. C. — Journaling the Linux ext2fs File System. USENIX Annual Technical Conference, 1998. The paper that introduced the journaling layer that became ext3's JBD.
  • Prabhakaran, V., Arpaci-Dusseau, A. C., & Arpaci-Dusseau, R. H. — Analysis and Evolution of Journaling File Systems. USENIX Annual Technical Conference, 2005. A systematic analysis of the journaling modes and their performance.
  • Rosenblum, M., & Ousterhout, J. K. — The Design and Implementation of a Log- Structured File System. ACM Transactions on Computer Systems, 1992. The paper that introduced log-structured file systems.
  • Rodeh, O., Bacik, J., & Mason, C. — BTRFS: The Linux B-Tree Filesystem. ACM Transactions on Storage, 2013. The design of Btrfs's copy-on-write consistency model.
  • Kerrisk, M. — The Linux Programming Interface, Chapter 14, File Systems. The fsync, fdatasync, and sync_file_range interfaces and their role in durability.
  • Linux man pages: man 2 fsync, man 2 fdatasync, man 2 sync, man 8 tune2fs (for the has_journal, journal_data, and journal_data_ordered options), man 5 ext4 (for the journaling options), man 8 e2fsck, man 1 debugfs (for the journal inode), man 5 proc (for /proc/fs/jbd2/).

Reading strategy

Read OSTEP's chapter first. It develops the crash- consistency problem from a concrete example — a file append that updates an inode's size and a block bitmap — and shows step by step why a crash between the two writes leaves the file system inconsistent. Then read the Silberschatz section for the formal classification of recovery mechanisms (consistency checking, journaling, and log-structured file systems). The Tweedie paper is the primary source for ext3's design, and the Prabhakaran et al. paper is the systematic analysis of the journaling modes' performance and consistency properties. Finally, the Btrfs paper is the modern view: it shows how a copy-on-write file system achieves consistency without a journal, and it is the bridge to Lesson 10.7. Pay particular attention to the difference between consistency (the file system's metadata is internally coherent after recovery) and durability (the data that an application wrote is actually on stable storage). Journaling provides consistency; it does not, by itself, provide durability. The two are often confused, and the confusion is the source of many bugs in applications that assume write is durable.

04 Core content

4.1 The crash-consistency problem

A file system's metadata is a set of structures that refer to one another. The superblock records where the inode table and the block bitmaps are; the inode table records each file's size, its owner, and its block pointers; the block bitmap records which blocks are free; the directory entries map names to inode numbers. An operation that modifies a file often modifies several of these structures, and the modifications must be applied together to keep the structures consistent.

Consider a simple operation: appending a block to a file. The file system must:

  1. Find a free block and mark it as allocated in the block bitmap.
  2. Update the inode's block pointer to refer to the new block.
  3. Update the inode's size field to include the new block's data.
  4. Update the superblock's free-block count.

Each of these steps is a write to a different block on the device. If the system crashes after step 1 but before step 2, the block bitmap says the block is allocated, but no inode points to it. The block is leaked: it can never be used again, and the file system has lost that space. If the system crashes after step 2 but before step 4, the free- block count is too high, and a later allocation may claim a block that is already in use. The file system is inconsistent: its structures contradict one another.

The inconsistency is not necessarily catastrophic. A leaked block wastes space; a wrong free-block count causes a future allocation to overwrite a file's data. But the file system cannot tolerate either state indefinitely, because the inconsistencies accumulate: each crash adds a few more, and after many crashes the file system's structures are so contradictory that it cannot be mounted. The file system must therefore have a mechanism for repairing the inconsistencies after a crash.

Crash consistency is not the same as durability

Consistency means that the file system's metadata is internally coherent: every allocated block is referenced by exactly one file, every directory entry refers to an inode that exists, and every free block is genuinely free. Durability means that the data an application wrote has reached stable storage and will survive a crash. The two are independent. A file system can be consistent but not durable (the metadata is coherent, but the file's data was never written to disk). A file system can be durable but not consistent (the data reached the disk, but the metadata update that describes it did not). Journaling addresses consistency; durability requires the application to call fsync or fdatasync to force the data and the metadata to the device.

4.2 The classical response: fsck

The classical recovery mechanism is the file system consistency check, known as fsck (on Linux, the fsck command and its file-system-specific variants such as e2fsck for ext2/ext3/ext4 and xfs_repair for XFS). fsck is a program that scans the file system's metadata, cross-checks the structures against one another, and repairs the inconsistencies it finds.

fsck's checks are systematic:

  • Superblock: verify the file system's parameters (block size, inode count, free-block count) and compare the free-block count with the number of free blocks in the bitmap.
  • Inodes: for each inode, verify that its block pointers refer to allocated blocks, that its size is consistent with its block count, and that its link count matches the number of directory entries that refer to it.
  • Block bitmap: build a reference count for every block by scanning all the inodes, and compare the result with the bitmap. A block that is referenced by a file but marked free, or marked allocated but referenced by no file, is an inconsistency.
  • Directories: verify that each directory entry refers to an inode that exists, that the entry's name is valid, and that the directory's . and .. entries are correct.
  • Link counts: for each inode, count the directory entries that refer to it and compare the count with the inode's link count. A mismatch means the file system has lost or gained a reference.

fsck's repair options are chosen by the administrator: it can fix the inconsistencies automatically (the default in unattended mode), ask for confirmation (the default in interactive mode), or report them without fixing (the -n option). For severe inconsistencies, fsck may place the affected files in a lost+found directory, so the administrator can recover them manually.

fsck's problem is its cost. To check the link counts, it must scan every inode and every directory entry. To check the block bitmap, it must scan every inode's block pointers. The scan's cost is proportional to the file system's size and to the number of files, and on a large file system it is enormous: a 1 TB file system with a million files can take hours to check. During the check, the file system is unavailable, so the system is down for the duration. The cost is the reason file systems moved to journaling: a journal lets the file system recover in seconds, without a full scan.

4.3 Write-ahead logging and the journal

The journal (also called the write-ahead log or intent log) is a dedicated region of the file system that records the metadata changes a file system is about to make, before it makes them. The protocol is called write-ahead logging: the log entry is written before the change it describes. After a crash, the file system reads the log and replays the changes, bringing the metadata to a consistent state.

The protocol's correctness rests on a simple observation: if the log entry is on disk, the change it describes can be redone; if the log entry is not on disk, the change was never committed, and it can be discarded. The log is therefore the authoritative record of what the file system intended to do, and the metadata blocks are the derived state that can be reconstructed from the log.

The write-ahead logging protocol has four steps:

  1. Write the log entry. The file system writes the intended metadata changes to the journal. The entry includes the block numbers that will be modified and the new contents of those blocks.
  2. Write the commit record. When the transaction's log entries are all on disk, the file system writes a commit record to the journal. The commit record is a marker that says "this transaction is complete; it may be replayed."
  3. Apply the changes (checkpoint). The file system writes the metadata changes to their final locations on disk. This step is called checkpointing. It can be done lazily, because the journal's entry is the authoritative record until the checkpoint completes.
  4. Free the journal space. When the checkpoint is complete, the journal space occupied by the transaction can be reused. The journal is a circular log, so the space is reused in order.

The order of the steps is what makes the protocol correct. The commit record is written after the log entries, so a commit record on disk implies that all the transaction's log entries are also on disk. The metadata changes are written after the commit record, so a crash before the checkpoint leaves the journal with a complete transaction that can be replayed. A crash after the checkpoint leaves the journal with a transaction that has already been applied; replaying it again is harmless because the changes are idempotent (rewriting a block with the same contents leaves it unchanged).

The protocol's guarantee is atomicity: a transaction is either fully applied or not applied at all. The journal does not guarantee that every transaction is durable (that every committed transaction survives a crash); it guarantees that the file system's metadata is consistent after recovery, whether the last transaction was applied or not. The distinction is important: journaling gives consistency, and durability requires explicit fsync calls from the application.

The journal as a circular log and the write-ahead logging protocol A diagram with three parts. The top part shows the journal as a circular log containing transaction records, each with a transaction-begin record, metadata blocks, and a commit record, plus a checkpoint pointer showing how far the metadata has been applied. The middle part shows the write-ahead logging protocol as four ordered steps: write log entries, write commit record, checkpoint to final location, free journal space. The bottom part shows recovery: committed transactions are replayed, incomplete transactions are discarded. THE JOURNAL AS A CIRCULAR LOG Tx 1 begin, meta blocks, commit ✓ Tx 2 begin, meta blocks, commit ✓ Tx 3 begin, meta blocks, commit ✗ (crash) free journal space (reused in order) checkpoint (metadata applied up to here) WRITE-AHEAD LOGGING PROTOCOL 1 Write log Write the metadata changes to the journal. 2 Write commit Write the commit record after all log entries. 3 Checkpoint Apply the changes to their final locations. 4 Free space Reuse the journal space. RECOVERY AFTER A CRASH Committed transactions → replay Tx 1 commit ✓ Tx 2 commit ✓ Replay: rewrite the metadata blocks The changes are idempotent, so replay is safe even if some were already applied. Incomplete transactions → discard Tx 3 no commit ✗ (free space) reused on next write Discard: the transaction was never committed The file system returns to the last consistent state.
Figure 10.6 — The journal and the write-ahead logging protocol. The journal is a circular log that holds transactions, each consisting of a begin record, the metadata blocks that the transaction will modify, and a commit record. The commit record is written after all the transaction's log entries, so a commit record on disk implies that the entire transaction is recoverable. After a crash, the file system scans the journal: committed transactions are replayed (their metadata changes are re-applied to their final locations), and incomplete transactions (those without a commit record) are discarded. The checkpoint pointer records how far the metadata has been applied, so replay can begin at the checkpoint rather than at the journal's beginning.

4.4 The journal's structure

The journal is a circular log, divided into transactions. A transaction is a group of metadata changes that must be applied atomically — all or nothing. Each transaction has a begin record (a marker that starts the transaction), a sequence of log entries (the metadata blocks that the transaction will modify), and a commit record (a marker that says the transaction is complete and may be replayed).

The journal's records also include:

  • Descriptor blocks that list the block numbers the transaction will modify. The descriptor is what allows the recovery process to know which blocks to rewrite and where to write them.
  • Sequence numbers that order the transactions and allow the recovery process to detect the journal's wrap-around (the point at which new entries overwrite old ones).
  • Checksums that detect corruption in the journal's records. A record whose checksum does not match is treated as if it were not written at all — the transaction is discarded.

The journal's size is a trade-off. A larger journal can hold more transactions before it must be reused, which reduces the frequency of the checkpoint operations and the risk of running out of space under heavy write load. A smaller journal uses less disk space but forces more frequent checkpoints. On ext4, the journal's default size is 128 MB or the file system's size divided by 64, whichever is smaller; the size can be set at file system creation time (with mke2fs -J size=) and inspected with tune2fs -l.

4.5 Checkpointing and journal space reuse

The journal is a circular log, so its space is finite and must be reused. The file system cannot reuse a transaction's journal space until the transaction's changes have been applied to their final locations — a process called checkpointing. The checkpoint pointer is a marker that records how far the checkpointing has progressed: transactions before the pointer have been applied and their space can be reused; transactions after the pointer have not been applied and their space must be preserved.

Checkpointing can be done eagerly (immediately after the commit) or lazily (in the background, as the journal fills). The choice is a performance trade-off:

  • Eager checkpointing applies the changes immediately after the commit, so the journal space is freed quickly and the journal is rarely full. But the eager writes compete with the application's writes, so the application sees more latency.
  • Lazy checkpointing applies the changes in the background, so the application's writes are not delayed by the checkpoint. But the journal space is freed slowly, and if the journal fills before the checkpoint completes, the file system must either stall the application (to let the checkpoint catch up) or force an eager checkpoint.

In practice, most file systems use a combination: checkpointing runs in the background under normal conditions, but the file system forces a checkpoint when the journal's free space falls below a threshold. The result is that the application sees low latency under normal conditions, and the journal does not overflow under heavy write load.

4.6 Journaling modes

The write-ahead logging protocol describes how to log metadata changes. But a file system's operations modify both metadata (the inode, the bitmap, the directory) and data (the file's contents), and the order in which the metadata and data are written affects both the strength of the consistency guarantee and the performance. The file system therefore offers a choice of journaling modes.

ext3 and ext4 define three modes, which are the classical options:

Writeback (the weakest, the fastest)
Only the metadata is journaled. The data blocks are written to their final locations independently of the journal, and the file system does not order the data writes relative to the metadata writes. The result is that the metadata is always consistent after recovery, but the data may be stale: a file whose inode says it is 10 KB long may contain 10 KB of data that was written before the last write, or 5 KB of new data followed by 5 KB of old data. Writeback is the fastest mode because it journals only the metadata, but it gives the weakest consistency guarantee for file contents.
Ordered (the default, the balanced choice)
Only the metadata is journaled, but the data blocks are written before the metadata that describes them. The ordering ensures that when the metadata says a block belongs to a file, the block's data is already on disk (or at least has been written before the metadata). The result is that a file's contents are consistent with its metadata after recovery: a file whose inode says it is 10 KB long contains the correct 10 KB of data, even if the last write was interrupted. Ordered mode is the default on ext3 and ext4 because it gives a strong consistency guarantee for file contents at a modest performance cost.
Data (the strongest, the slowest)
Both the metadata and the data are journaled. Every block that a transaction modifies — metadata or data — is written to the journal before it is written to its final location. The result is that the file system can recover both the metadata and the data, so a file's contents are consistent and up to date after recovery. Data mode gives the strongest guarantee, but it is the slowest because every data block is written twice: once to the journal and once to its final location. The performance cost is significant for write-heavy workloads (the write throughput can be halved), so data mode is rarely used in practice.

The modes' trade-offs are summarized in the following table. The table's "data" column describes what happens to the file's contents if the system crashes after a write but before the next checkpoint.

Journaling modes on ext3/ext4
Mode What is journaled Data ordering File contents after crash Performance
Writeback Metadata only None Stale or partially updated; metadata is consistent Fastest
Ordered Metadata only Data before metadata Consistent with the metadata; the last write may be lost Balanced (default)
Data Metadata and data Data and metadata both journaled Consistent and up to date Slowest

The modes are set at mount time (with the data= mount option) or as the file system's default (with tune2fs -o). On ext4, the default is data=ordered. Other file systems use different names for the same concepts: XFS journals only metadata (there is no data journaling mode), and Btrfs achieves consistency through copy-on-write rather than through a journal.

Journaling does not make writes durable

A common misconception is that journaling makes an application's writes durable — that a write system call returns only after the data is on stable storage. This is not true. A write returns as soon as the data is in the kernel's page cache, which is memory, not disk. The data reaches the disk only when the kernel flushes the page cache, which may be seconds later. If the system crashes in between, the data is lost — even on a journaling file system. To make a write durable, the application must call fsync (or fdatasync) after the write, which forces the data and the metadata to the device. The journal protects the file system's metadata from corruption; it does not protect an application's data from loss unless the application explicitly asks for durability.

4.7 JBD2: ext3 and ext4's journaling layer

The journaling layer used by ext3 and ext4 is called JBD2 (the Journaling Block Device, version 2). JBD2 is a generic layer: it manages the journal's structure, the transactions, the commit records, and the recovery, but it is independent of the file system's specific metadata. The file system calls JBD2's interface to start a transaction, to add blocks to it, and to commit it, and JBD2 handles the write-ahead logging protocol and the recovery.

JBD2's design is worth examining because it illustrates the journaling layer's responsibilities:

  • Transaction management. JBD2 groups the file system's metadata changes into transactions. A transaction can include many changes from many operations, so the commit record is amortized over a large group of changes, which improves the journal's efficiency.
  • Journal space management. JBD2 maintains the journal's circular log, the checkpoint pointer, and the free space. It forces a checkpoint when the journal's free space falls below a threshold.
  • Recovery. After a crash, JBD2 scans the journal from the last checkpoint, identifies the committed transactions, and replays them. The replay is idempotent, so a transaction that was partially checkpointed before the crash is re-applied harmlessly.
  • Ordering. In ordered mode, JBD2 ensures that the data blocks are written before the metadata blocks that describe them. It does this by tracking the data blocks that a transaction depends on and writing them to disk before the commit record.

On ext4, JBD2's journal is inode 8 of the file system. The journal is itself a file — a circular file whose contents are the journal's records — and it can be examined with debugfs:

# Show the journal's inode
sudo debugfs -R 'stat <8>' /dev/sda1 2>/dev/null

# The output shows the journal's size, its blocks,
# and its extent tree.

# Show the journal's superblock (the JBD2 superblock)
sudo debugfs -R 'logdump -S' /dev/sda1 2>/dev/null

The journal's inode's size is the journal's size in bytes, and its block pointers describe the journal's location on the device. The JBD2 superblock records the journal's parameters, including the first and last transaction sequence numbers and the journal's block size.

4.8 Recovery

The recovery process after a crash is a scan of the journal. The file system:

  1. Reads the journal's superblock. The superblock records the sequence number of the first transaction that has not been checkpointed, which is where the scan begins.
  2. Scans the transactions. Starting at the first un-checkpointed transaction, the file system reads each transaction's begin record, its descriptor, its log entries, and its commit record.
  3. Identifies committed transactions. A transaction with a valid commit record and valid checksums is committed: its changes were fully written to the journal before the crash. A transaction without a valid commit record (or with an invalid checksum) is incomplete: its changes were not fully written, so it must be discarded.
  4. Replays the committed transactions. For each committed transaction, the file system writes the logged blocks to their final locations. The writes are idempotent, so replaying a transaction that was already partially applied is safe.
  5. Discards the incomplete transactions. The incomplete transactions' journal space is marked as free and is reused by future transactions.
  6. Marks the file system as clean. The superblock's "clean" flag is set, so the next mount does not require a recovery.

The recovery's cost is proportional to the journal's size, not to the file system's size. A typical journal is a few hundred megabytes, so the recovery takes a few seconds — orders of magnitude faster than a full fsck. The recovery's cost is the journaling file system's most important advantage: it is what makes a large file system practical, because a crash does not require a multi-hour fsck.

The recovery process is not perfect. It can only replay the transactions that were in the journal at the time of the crash; if the journal is corrupted, the recovery fails and the file system must fall back to a full fsck. The file system's superblock therefore records a mount count and a clean flag: if the file system was not unmounted cleanly, the next mount runs the recovery; if the recovery fails, the mount fails and the administrator must run e2fsck manually.

4.9 Alternatives to journaling

Journaling is the most common crash-consistency mechanism, but it is not the only one. Two alternatives are worth examining: log-structured file systems and copy-on-write file systems.

A log-structured file system (LFS) writes all of its data — metadata and file contents — to a single log that is the only structure on the device. The log is written sequentially, so the file system never seeks to update a block in place; it appends the new version of the block to the log. The log's head is the file system's current state, and the log's tail is the oldest data, which is garbage-collected when its blocks are no longer referenced. LFS's advantage is its write performance: because all writes are sequential, LFS achieves the device's full sequential bandwidth even for random updates. Its disadvantage is the garbage collection's cost and complexity, and the difficulty of finding the current version of a block (the file system must maintain an index that maps block numbers to their current locations in the log). LFS's ideas influenced the design of flash file systems (which are inherently log-structured, because flash memory cannot be updated in place) and of modern copy-on-write file systems.

A copy-on-write (CoW) file system, such as Btrfs or ZFS, achieves crash consistency by never overwriting a block in place. When a block is modified, the file system writes the new version to a new location and updates the block's parent to point to the new version. The update propagates up the tree of metadata, and the final write is the superblock's root pointer, which is updated atomically (a single block write). If a crash occurs before the root pointer's update, the file system's state is the old state, because the new blocks are unreferenced and the old tree is intact. The result is that CoW file systems are always consistent: there is no need for a journal, because the atomic root-pointer update is the only change that must be crash-safe.

CoW's advantages are the absence of a journal (so no journal writes to pay for), the ability to take cheap snapshots (a snapshot is just a copy of the root pointer), and the ability to detect corruption (each block has a checksum, and the tree's structure allows the file system to verify that the checksums are consistent). Its disadvantages are the write amplification (a single logical write can cause several physical writes as the tree is updated), the fragmentation (the new blocks are allocated in new locations, so the file system must manage the free space carefully), and the complexity of the garbage collection (old versions of blocks must be freed when they are no longer referenced). Btrfs and ZFS are the modern examples; they are increasingly popular for their data integrity features and their snapshot capabilities, and they are the subject of Lesson 10.7.

Journaling, LFS, and CoW: three routes to consistency

The three mechanisms achieve crash consistency by different routes. Journaling keeps the file system's structures in place and uses a separate log to record the changes before they are applied. LFS uses the log itself as the file system's only structure, so there is no separate journal. CoW never overwrites a block in place, so the file system's state changes only when the root pointer is updated. The three mechanisms have different performance profiles and different failure modes, but they all provide the same basic guarantee: after a crash, the file system is in a consistent state. The choice between them is a matter of the workload, the hardware, and the file system's other features (snapshots, checksums, RAID).

4.10 Checksums and the limits of journaling

Journaling protects against crashes, but it does not protect against corruption. If a block is corrupted by a failing disk, a cosmic ray, or a firmware bug, the journal's replay may write the corrupted block's contents to its final location, and the file system's metadata will be corrupt despite the journal. The journal detects corruption in its own records (through the records' checksums), but it does not detect corruption in the metadata blocks that it writes.

The modern response is metadata checksums: every metadata block includes a checksum, and the file system verifies the checksum when it reads the block. If the checksum does not match, the file system knows the block is corrupt and can take corrective action — refusing to use the block, reporting the corruption to the administrator, or (on a file system with redundancy, such as ZFS or Btrfs) recovering the block from a replica.

ext4, XFS, and Btrfs all support metadata checksums. On ext4, the feature is called metadata_csum and is enabled by default on new file systems; on Btrfs and ZFS, checksums are fundamental to the file system's design, and every block (metadata and data) has a checksum. The checksums are what turn a silent corruption into a detected corruption, and detection is the first step toward recovery. Without checksums, a corrupted metadata block is indistinguishable from a valid one, and the file system may propagate the corruption through its structures before anyone notices.

The combination of journaling and checksums is the modern file system's standard defence: the journal provides crash consistency, and the checksums provide corruption detection. The two mechanisms are complementary, and a file system that has only one of them is vulnerable to the failure mode that the other addresses.

4.11 Common misconceptions

"Journaling makes the file system slow."

Journaling adds writes (the journal entries) to every metadata update, but it also allows the file system to defer the metadata writes and to batch them, which can improve performance under heavy load. In ordered mode, the data ordering adds some latency, but the overhead is small for most workloads. Journaling's benefit — fast recovery — usually outweighs its cost.

"Journaling makes my writes durable."

Journaling makes the file system's metadata consistent, not the application's data durable. A write returns as soon as the data is in the page cache, which is memory. To make the data durable, the application must call fsync or fdatasync. Even on a journaling file system, a crash between the write and the flush loses the data.

"Data mode is always the safest choice."

Data mode journals both metadata and data, which gives the strongest guarantee, but it writes every data block twice (once to the journal, once to its final location), which can halve the write throughput. Ordered mode gives a strong guarantee for most purposes (the file's contents are consistent with its metadata) at a much lower cost, which is why it is the default.

"fsck is obsolete."

fsck is still needed. It is the fallback when the journal is corrupted or when the file system has inconsistencies that the journal cannot repair (e.g., a disk error that corrupts a metadata block). Journaling reduces the frequency of fsck, not its necessity. A well-designed file system uses the journal for the common case and fsck for the rare one.

"The journal stores the file's data."

In the default ordered mode, the journal stores only metadata. The file's data is written to its final location, in order relative to the metadata. Only in data mode does the journal store the file's data as well, and data mode is rarely used because of its performance cost.

"A journaling file system never needs a check."

A journaling file system still benefits from periodic checks. The journal handles crashes, but it does not detect corruption, and the file system's other structures (the free-space bitmaps, the inode tables) can drift out of sync over time. Periodic offline checks (e.g., e2fsck on an unmounted file system) catch problems that the journal does not.

"Copy-on-write file systems do not need a journal."

Correct — but they do need a mechanism for the atomic root-pointer update. CoW file systems avoid the journal by making the superblock's root pointer the only structure that must be updated atomically, and a single block write is atomic on most devices. The result is that CoW file systems achieve crash consistency without a journal, but they pay for it with write amplification and garbage collection.

"Checksums and journaling are the same thing."

They are different. Journaling provides crash consistency: after a crash, the file system's metadata is coherent. Checksums provide corruption detection: when a block is read, the file system verifies that its contents match the stored checksum. The two mechanisms address different failure modes (crashes versus corruption), and a file system needs both to be robust.

05 Worked example

The following example traces a file-creation operation through the journaling modes, computes the cost of each mode, and demonstrates the recovery process.

Tracing a file creation through the journal

Consider a file-creation operation on an ext4 file system. The operation creates a new file report.txt in the directory /home/user and writes 4 KB of data to it. The operation modifies the following metadata blocks:

  • The inode bitmap of the block group that contains the new inode: one bit is set to mark the inode as allocated.
  • The inode table block that contains the new inode: the inode is initialized with the file's metadata (mode, owner, size, block pointers).
  • The block bitmap of the block group that contains the new data block: one bit is set to mark the block as allocated.
  • The directory block that contains the directory entry: a new entry with the name report.txt and the new inode number is added.
  • The superblock: the free-inode count and the free-block count are decremented.

The operation therefore modifies five metadata blocks, plus the file's data block. The file system groups the metadata changes into a transaction, and the journaling mode determines what is journaled and in what order.

Writeback mode

In writeback mode, only the metadata blocks are journaled. The file system:

  1. Writes the five metadata blocks to the journal, with a begin record and a descriptor that lists their block numbers.
  2. Writes the commit record.
  3. Writes the data block to its final location (asynchronously, whenever the page cache is flushed).
  4. Checkpoints the five metadata blocks to their final locations.

If the system crashes after the commit but before the checkpoint, the recovery replays the metadata blocks. The file system is consistent: the inode exists, the directory entry refers to it, the bitmaps are correct. But the file's data block may contain old data, because the data block's write was not ordered relative to the metadata. A program that reads the file after the crash sees the correct metadata (the file is 4 KB long) but possibly incorrect contents.

Ordered mode (the default)

In ordered mode, only the metadata blocks are journaled, but the data block is written before the metadata that describes it. The file system:

  1. Writes the data block to its final location.
  2. Writes the five metadata blocks to the journal, with a begin record and a descriptor.
  3. Writes the commit record.
  4. Checkpoints the five metadata blocks to their final locations.

The ordering is the crucial difference. The data block is on disk before the metadata says it belongs to the file, so a crash after the commit leaves the file with its correct contents. The file system is consistent, and the file's contents are consistent with its metadata. This is the strong guarantee that ordered mode provides, and it is why ordered mode is the default.

Data mode

In data mode, both the metadata blocks and the data block are journaled. The file system:

  1. Writes the data block and the five metadata blocks to the journal, with a begin record and a descriptor.
  2. Writes the commit record.
  3. Checkpoints the six blocks to their final locations.

The file system writes the data block twice: once to the journal and once to its final location. The write amplification is 2× for the data block, which is why data mode is the slowest. Its advantage is that the recovery replays the data block as well as the metadata, so a crash after the commit leaves the file's contents up to date.

Cost comparison

The following table compares the cost of the three modes for the file-creation operation. The costs are in disk writes (the number of blocks written).

Mode Metadata writes Data writes Total writes Consistency
Writeback 5 (journal) + 5 (checkpoint) = 10 1 11 Metadata only
Ordered 5 (journal) + 5 (checkpoint) = 10 1 11 Metadata + data ordering
Data 5 (journal) + 5 (checkpoint) = 10 1 (journal) + 1 (final) = 2 12 Metadata + data

The table shows that the three modes have similar metadata costs (10 writes each, because the metadata is always journaled and then check-pointed) and differ only in the data writes. Ordered mode writes the data block once, before the metadata; data mode writes it twice. Writeback writes it once but does not order it. The difference between ordered and data mode is one extra data write per transaction, which is small for this example but becomes significant for write-heavy workloads.

Recovery

Suppose the system crashes during the checkpoint, after the commit record is written but before all the metadata blocks are applied. On the next mount, the file system:

  1. Reads the journal's superblock and finds the first un-checkpointed transaction.
  2. Scans the transaction, finds its begin record, its descriptor, its log entries, and its commit record.
  3. Verifies the commit record and the log entries' checksums.
  4. Replays the transaction by writing the five metadata blocks to their final locations. The blocks that were already checkpointed before the crash are written again with the same contents, which is harmless (idempotent).
  5. Marks the transaction as checkpointed and frees its journal space.
  6. Marks the file system as clean.

The recovery takes a few seconds, because it reads only the journal (a few hundred megabytes at most) rather than the entire file system. If the system had no journal, the recovery would require a full fsck, which on a large file system could take hours.

Observing the journal on a live system

The Linux tools let you examine the journal's parameters and its contents. The following commands show the journal's size, its mode, and its inode.

# Show the file system's journal parameters
sudo tune2fs -l /dev/sda1 2>/dev/null | grep -i journal

# The output shows the journal's inode number (8),
# its size, and its mode (e.g., "ordered").

# Show the journal's inode
sudo debugfs -R 'stat <8>' /dev/sda1 2>/dev/null

# Show the journal's superblock
sudo debugfs -R 'logdump -S' /dev/sda1 2>/dev/null

# Show the JBD2 statistics in /proc
cat /proc/fs/jbd2/sda1-8/info 2>/dev/null

The tune2fs output shows the journal's mode (the data= option), its size, and its inode number. The debugfs output shows the journal's inode and its blocks. The /proc/fs/jbd2/ directory shows the journal's statistics, including the number of transactions, the number of commits, and the checkpoint's progress.

06 Interactive activity

Activity · explore journaling on a live system

The goal of this activity is to use the Linux command line to observe a file system's journaling configuration, to measure the effect of the journaling modes, and to demonstrate the recovery process. Work through the steps in your course VM.

  1. Inspect the journal's configuration. Use tune2fs to see the journal's parameters.
    # Show the file system's features and journal parameters
    sudo tune2fs -l /dev/sda1 2>/dev/null | grep -iE 'journal|features|mount'
    
    # The output shows whether the file system has a journal
    # (the 'has_journal' feature), the journal's inode number,
    # and the journal's mode.
    What is the journal's mode? Is the journal enabled (has_journal)? What is the journal's inode number? Write down what you find.
  2. Examine the journal's inode. Use debugfs to examine the journal's inode (inode 8 on ext4).
    # Show the journal's inode
    sudo debugfs -R 'stat <8>' /dev/sda1 2>/dev/null
    
    # The output shows the journal's size, its block count,
    # and its extent tree.
    How large is the journal? How many blocks does it occupy? What is the journal's inode's mode (is it a regular file, a special file)? Explain what you observe.
  3. Examine the journal's superblock. Use debugfs to examine the JBD2 superblock.
    # Show the JBD2 superblock
    sudo debugfs -R 'logdump -S' /dev/sda1 2>/dev/null
    The output shows the journal's first and last transaction sequence numbers, the journal's block size, and other parameters. What is the journal's block size? How many transactions can the journal hold?
  4. Observe the journal's activity. Use the /proc/fs/jbd2/ directory to observe the journal's statistics while a workload runs.
    # Show the journal's statistics
    cat /proc/fs/jbd2/sda1-8/info 2>/dev/null
    
    # Run a write-heavy workload
    for i in $(seq 1 1000); do
        echo "test $i" > /tmp/journal-test-$i.txt
    done
    
    # Show the statistics again
    cat /proc/fs/jbd2/sda1-8/info 2>/dev/null
    The statistics include the number of transactions, the number of commits, the number of blocks journaled, and the checkpoint's progress. How do the counters change after the workload? What does the change tell you about the journal's behaviour?
  5. Measure the effect of the journaling mode. If you have permission to remount the file system (in a VM, you usually do), change the journaling mode and measure the effect on a write-heavy workload.
    # Show the current mode
    mount | grep ' / '
    sudo tune2fs -l /dev/sda1 2>/dev/null | grep -i 'journal data'
    
    # Remount with a different mode (this requires root)
    sudo mount -o remount,data=writeback /
    mount | grep ' / '
    
    # Run a write-heavy workload
    time for i in $(seq 1 10000); do
        echo "test" > /tmp/mode-test-$i.txt
    done
    
    # Remount with the default mode
    sudo mount -o remount,data=ordered /
    
    # Clean up
    rm -f /tmp/mode-test-*.txt
    How does the writeback mode's performance compare to the ordered mode's? Is the difference significant? Why? Remember that the difference is in the data ordering, not in the metadata journaling, so the effect is largest for workloads with many small data writes.
  6. Observe the recovery process. The recovery process runs when the file system is mounted after an unclean shutdown. You can simulate this by creating a file, forcing a write, and then examining the journal's state.
    # Create a file and force the write
    echo "recovery test" > /tmp/recovery-test.txt
    sync
    
    # Examine the journal's superblock
    sudo debugfs -R 'logdump -S' /dev/sda1 2>/dev/null
    
    # The journal's sequence numbers show the last committed
    # and checkpointed transactions. After a crash, the file
    # system would replay the transactions between the
    # checkpoint and the last commit.
    What are the journal's first and last sequence numbers? What do they tell you about the journal's state? If the system crashed now, which transactions would be replayed?
  7. Compare journaling and no journaling. If your VM has multiple file systems, compare a journaling file system (ext4) with a non- journaling one (ext2, if available).
    # Check the file system types
    df -T
    
    # If ext2 is available, create a small ext2 file system
    # in a file and mount it
    dd if=/dev/zero of=/tmp/ext2.img bs=1M count=100
    sudo mkfs.ext2 /tmp/ext2.img
    sudo mkdir -p /mnt/ext2
    sudo mount -o loop /tmp/ext2.img /mnt/ext2
    
    # Compare the mount time and the performance
    time (sudo umount /mnt/ext2 && sudo mount -o loop /tmp/ext2.img /mnt/ext2)
    How does the ext2 file system's mount time compare to ext4's? What happens if you simulate a crash by not unmounting the file system before rebooting? (Do not actually do this on your VM's root file system.)
  8. Read the man pages. Read the man pages that document the journaling interface.
    man 2 fsync
    man 2 fdatasync
    man 2 sync
    man 5 ext4
    man 8 tune2fs
    man 8 e2fsck
    For each, write a sentence describing its main contribution to your understanding of journaling. The man 2 fsync page documents the durability interface; the man 5 ext4 page documents the journaling options; the man 8 tune2fs page documents the tool that changes the journal's parameters.

Reflect: journaling is one of the most successful ideas in file system design. It turned the crash-recovery problem from a multi-hour, error-prone scan into a few seconds of deterministic replay, and it made large file systems practical. But journaling is not a complete solution: it protects metadata, not data; it does not detect corruption; and its performance cost is real. The modern file systems combine journaling with checksums, copy-on-write, and other mechanisms to address the failure modes that journaling alone cannot. Which of these mechanisms matter most for the workloads you care about? What would you sacrifice — performance, durability, or simplicity — to get the strongest guarantee? How would you explain the difference between consistency and durability to a programmer who assumes that write means "saved"?

07 Knowledge check

Check your understanding

Question 1

Explain the crash-consistency problem. Why does a multi-block metadata update leave the file system vulnerable to a crash, and what is the classical response?

Show answer
The crash-consistency problem is the problem of keeping a file system's on-disk metadata consistent when a crash interrupts an update. A file system's metadata is a set of structures that refer to one another: the superblock records where the inode table and the block bitmaps are, the inode table records each file's block pointers, the bitmaps record which blocks are free, and the directory entries map names to inode numbers. An operation that modifies a file often modifies several of these structures, and the modifications must be applied together to keep the structures consistent. Consider an append: the file system must mark a free block as allocated in the block bitmap, update the inode's block pointer, update the inode's size, and decrement the superblock's free-block count. Each of these is a separate disk write. If the system crashes after the first write but before the last, the file system is left in a state that is neither the old state nor the new one: a block may be marked allocated but referenced by no file (a leak), or the free-block count may be too high (a future allocation may claim a block that is in use). The file system is inconsistent. The classical response is fsck, the file system consistency check. fsck scans the file system's metadata, cross-checks the structures against one another (for example, building a reference count for every block by scanning all the inodes and comparing the result with the bitmap), and repairs the inconsistencies it finds. fsck is effective, but it is slow: it must read the entire file system, and its cost grows with the file system's size. On a large file system, fsck can take hours, during which the file system is unavailable. The cost of fsck is the reason file systems moved to journaling, which recovers in seconds by replaying a small log.

Question 2

Describe the write-ahead logging protocol. What are the protocol's steps, and why does the order of the steps matter?

Show answer
Write-ahead logging is the protocol on which journaling is based: the log entry is written before the change it describes. The protocol has four steps. Step 1: write the log entry. The file system writes the intended metadata changes to the journal. The entry includes a descriptor that lists the block numbers that will be modified and the new contents of those blocks. Step 2: write the commit record. When all the transaction's log entries are on disk, the file system writes a commit record, a marker that says the transaction is complete and may be replayed. Step 3: apply the changes (checkpoint). The file system writes the metadata changes to their final locations on disk. This step can be done lazily, because the journal's entry is the authoritative record until the checkpoint completes. Step 4: free the journal space. When the checkpoint is complete, the journal space occupied by the transaction can be reused. The order of the steps is what makes the protocol correct. The commit record is written after the log entries, so a commit record on disk implies that all the transaction's log entries are on disk. The metadata changes are written after the commit record, so a crash before the checkpoint leaves the journal with a complete transaction that can be replayed. A crash after the checkpoint leaves the journal with a transaction that has already been applied; replaying it again is harmless because the changes are idempotent. The protocol's guarantee is atomicity: a transaction is either fully applied or not applied at all. The journal does not guarantee durability (that every committed transaction survives a crash); it guarantees that the file system's metadata is consistent after recovery.

Question 3

Compare the three journaling modes (writeback, ordered, data). What does each journal, what is the ordering, what consistency guarantee does each provide, and what is the performance cost of each?

Show answer
The three modes differ in what is journaled and in the ordering of the data writes relative to the metadata writes. Writeback journals only the metadata and does not order the data writes relative to the metadata writes. The metadata is always consistent after recovery, but the file's contents may be stale or partially updated: a file whose inode says it is 10 KB long may contain 10 KB of old data, or 5 KB of new data followed by 5 KB of old data. Writeback is the fastest mode because it journals only the metadata. Ordered journals only the metadata, but it writes the data blocks before the metadata that describes them. The ordering ensures that when the metadata says a block belongs to a file, the block's data is already on disk. The result is that a file's contents are consistent with its metadata after recovery: a file whose inode says it is 10 KB long contains the correct 10 KB of data, even if the last write was interrupted. Ordered mode is the default on ext3 and ext4 because it gives a strong consistency guarantee for file contents at a modest performance cost. Data journals both the metadata and the data: every block that a transaction modifies is written to the journal before it is written to its final location. The result is that the file system can recover both the metadata and the data, so a file's contents are consistent and up to date after recovery. Data mode gives the strongest guarantee, but it is the slowest because every data block is written twice: once to the journal and once to its final location. The performance cost is significant for write-heavy workloads, so data mode is rarely used in practice. The three modes therefore trade consistency for performance: writeback is the fastest but weakest, ordered is the balanced default, and data is the strongest but slowest.

Question 4

Describe the recovery process after a crash on a journaling file system. How does the file system decide which transactions to replay and which to discard, and why is the recovery fast?

Show answer
The recovery process is a scan of the journal. The file system reads the journal's superblock, which records the sequence number of the first transaction that has not been checkpointed, and begins the scan there. It reads each transaction's begin record, its descriptor, its log entries, and its commit record, and it verifies the commit record and the log entries' checksums. A transaction with a valid commit record and valid checksums is committed: its changes were fully written to the journal before the crash, so it can be replayed. A transaction without a valid commit record (or with an invalid checksum) is incomplete: its changes were not fully written, so it must be discarded. The file system replays each committed transaction by writing the logged blocks to their final locations. The writes are idempotent, so replaying a transaction that was already partially applied is safe: writing the same contents to a block that already has them leaves the block unchanged. After the replay, the file system marks the transactions as checkpointed, frees their journal space, and sets the superblock's clean flag. The incomplete transactions are discarded: their journal space is marked free and reused by future transactions. The recovery is fast because its cost is proportional to the journal's size, not to the file system's size. A typical journal is a few hundred megabytes, so the recovery reads a few hundred megabytes and writes a few hundred megabytes — a few seconds of I/O. A full fsck, by contrast, must read the entire file system, which on a large file system can be terabytes, and it must cross-check every structure, which takes hours. The journal's bounded size is what makes the recovery's cost bounded, and the bounded cost is journaling's most important advantage.

Question 5

Compare journaling, log-structured file systems, and copy-on-write file systems as mechanisms for crash consistency. What are the advantages and disadvantages of each?

Show answer
The three mechanisms achieve crash consistency by different routes. Journaling keeps the file system's structures in place and uses a separate log to record metadata changes before they are applied. Its advantage is that it is well understood, it is efficient for the common case (the metadata is written twice: once to the journal, once to its final location), and it recovers in seconds. Its disadvantages are that it journals metadata only in the default mode (so it does not protect file contents from stale-data problems), it does not detect corruption, and its performance cost is real for write-heavy workloads. Log- structured file systems (LFS) write all data — metadata and file contents — to a single log that is the only structure on the device. The log is written sequentially, so the file system never seeks to update a block in place, and it achieves the device's full sequential bandwidth even for random updates. Its advantages are its write performance and its natural fit with flash memory (which cannot be updated in place). Its disadvantages are the garbage collection's cost and complexity (the old versions of blocks must be freed) and the difficulty of finding the current version of a block (the file system must maintain an index that maps block numbers to their current locations). Copy-on-write (CoW) file systems, such as Btrfs and ZFS, never overwrite a block in place: when a block is modified, the new version is written to a new location and the block's parent is updated to point to it, and the update propagates up the tree until the superblock's root pointer is updated atomically. Its advantages are the absence of a journal (no journal writes to pay for), the ability to take cheap snapshots (a snapshot is a copy of the root pointer), and the ability to detect corruption (each block has a checksum, and the tree's structure allows the file system to verify consistency). Its disadvantages are write amplification (a single logical write can cause several physical writes as the tree is updated), fragmentation (the new blocks are allocated in new locations), and the complexity of the garbage collection. The three mechanisms all provide the same basic guarantee — after a crash, the file system is consistent — but they have different performance profiles and different failure modes, and the choice between them depends on the workload, the hardware, and the file system's other features.

08 Unit discussion

Discussion prompt

Journaling was introduced in the late 1980s and early 1990s as a response to the growing cost of fsck. In the 1980s, a file system was small enough that a full fsck took minutes; by the late 1990s, file systems had grown to the point where a full fsck took hours, and the downtime was unacceptable. Journaling solved the problem by making recovery proportional to the journal's size rather than the file system's size, and it became the standard mechanism for crash consistency on general-purpose file systems. The idea was not new — databases had used write-ahead logging since the 1970s — but its application to file systems was transformative.

Journaling's success has been so complete that it is easy to forget that it is a compromise. It provides consistency, not durability; it protects metadata, not data (in the default mode); it does not detect corruption; and it costs performance. The modern file systems have added mechanisms to address journaling's limitations: checksums to detect corruption, copy-on-write to avoid the journal's cost, and snapshots to provide point-in-time recovery. The result is a layered defence: the journal handles crashes, the checksums handle corruption, and the snapshots handle user error.

The trend is toward end-to-end data integrity: the file system verifies its own metadata with checksums, the application verifies its own data with checksums, and the two layers together detect corruption that neither could detect alone. The trend is a response to the observation that disks lie: a disk can report a successful write that never reached the platter, and it can return corrupted data without an error. The file system cannot trust the device, and the application cannot trust the file system. The checksums are the mechanism that turns silent corruption into detected corruption, and detection is the first step toward recovery.

Should file systems provide end-to-end integrity, or should that be the application's responsibility? What is the right division of labour between the file system and the application for durability and integrity? Should a file system's default configuration favour performance or safety, and who should decide? Post your response to the Unit 10 discussion forum and reply to at least two classmates. Supports CLO1, CLO5, and CLO6.

09 Practice exercise

This ungraded exercise consolidates the lesson and prepares you for Lesson 10.7. Aim to spend 60–90 minutes on it.

  1. Trace a metadata update. A file system performs a rename operation: a file old.txt is renamed to new.txt in the same directory. The operation modifies the directory block that contains the entry. Trace the operation under each of the three journaling modes (writeback, ordered, data), showing:
    • Which blocks are written to the journal.
    • The order of the writes.
    • The state of the file system after a crash at each step.
    Which mode gives the strongest guarantee for this operation? Why?
  2. Compare the modes. Write a short paragraph (200–250 words) comparing the three journaling modes. Your paragraph should address: (a) what each mode journals; (b) the data ordering; (c) the consistency guarantee for the file's metadata; (d) the consistency guarantee for the file's contents; (e) the performance cost; (f) the workloads for which each mode is best suited.
  3. Measure the effect of the journaling mode. On a test file system (a loop-mounted image, so you do not risk your VM's root file system), measure the throughput of a write-heavy workload under the three modes.
    # Create a test file system
    dd if=/dev/zero of=/tmp/test.img bs=1M count=500
    sudo mkfs.ext4 /tmp/test.img
    sudo mkdir -p /mnt/test
    sudo mount -o loop /tmp/test.img /mnt/test
    
    # Measure under ordered mode (the default)
    sudo mount -o remount,data=ordered /mnt/test
    time (for i in $(seq 1 1000); do
        dd if=/dev/zero of=/mnt/test/f$i bs=4k count=1 2>/dev/null
    done)
    
    # Measure under writeback mode
    sudo mount -o remount,data=writeback /mnt/test
    time (for i in $(seq 1 1000); do
        dd if=/dev/zero of=/mnt/test/f$i bs=4k count=1 2>/dev/null
    done)
    
    # Measure under data mode
    sudo mount -o remount,data=journal /mnt/test
    time (for i in $(seq 1 1000); do
        dd if=/dev/zero of=/mnt/test/f$i bs=4k count=1 2>/dev/null
    done)
    What are the three modes' throughputs? How do they compare? Does the ordering match your expectations (writeback fastest, data slowest)? Explain.
  4. Examine the journal's contents. Use debugfs to dump the journal's contents before and after a workload.
    # Dump the journal's contents
    sudo debugfs -R 'logdump' /dev/sda1 2>/dev/null | head -50
    
    # Run a workload that modifies metadata
    for i in $(seq 1 100); do
        echo "test" > /tmp/journal-test-$i.txt
    done
    
    # Dump the journal's contents again
    sudo debugfs -R 'logdump' /dev/sda1 2>/dev/null | head -50
    What is the difference between the two dumps? Can you identify the transaction records, the descriptor blocks, and the commit records? What does the journal's structure look like?
  5. Simulate a crash. On a test file system (a loop-mounted image), simulate a crash by mounting the file system, writing a file, and then forcibly unmounting it (or by using a VM snapshot and reverting it). Observe the recovery on the next mount.
    # Create a test file system
    dd if=/dev/zero of=/tmp/crash.img bs=1M count=200
    sudo mkfs.ext4 /tmp/crash.img
    sudo mount -o loop /tmp/crash.img /mnt/crash
    
    # Write a file
    echo "before crash" > /mnt/crash/test.txt
    sync
    
    # Force an unclean state by not unmounting (or by
    # using a VM snapshot). Then mount the file system again.
    sudo mount -o loop /tmp/crash.img /mnt/crash
    
    # Check the kernel's messages for the recovery
    dmesg | tail -20
    What does the kernel's message say about the recovery? How long did the recovery take? How does it compare to a full fsck?
  6. Investigate a CoW file system. If your VM supports Btrfs, create a Btrfs file system and compare its consistency mechanism to ext4's.
    # Create a Btrfs file system
    dd if=/dev/zero of=/tmp/btrfs.img bs=1M count=500
    sudo mkfs.btrfs /tmp/btrfs.img
    sudo mount -o loop /tmp/btrfs.img /mnt/btrfs
    
    # Take a snapshot
    sudo btrfs subvolume snapshot /mnt/btrfs /mnt/btrfs/snap1
    
    # Modify a file
    echo "modified" > /mnt/btrfs/test.txt
    
    # Compare the snapshot with the current state
    ls -la /mnt/btrfs/
    ls -la /mnt/btrfs/snap1/
    
    # Check the file system's checksum feature
    sudo btrfs filesystem df /mnt/btrfs
    How does Btrfs's snapshot capability compare to ext4's journaling? What does the snapshot demonstrate about the CoW approach?
  7. Design a recovery strategy. A system administrator is responsible for a database server with a 10 TB file system. The server must be highly available, and a crash must not require a multi-hour fsck. Design a recovery strategy for this system. Consider:
    • Which file system would you use, and why?
    • Which journaling mode would you choose, and why?
    • How would you handle the database's own durability requirements (the database's WAL)?
    • How would you verify the file system's integrity periodically?
    • How would you handle a disk failure that corrupts a metadata block?
    Justify your design.
  8. Reflect. In your own words, explain why journaling is necessary for a modern file system, and describe the trade-offs that the journaling modes make. Then describe one situation in your own experience — a program you have written or a system you have used — where durability or consistency mattered. What was the workload, and how would a different journaling mode or a different consistency mechanism have changed the outcome?

10 Summary

  • The crash-consistency problem is the problem of keeping a file system's on-disk metadata consistent when a crash interrupts an update. A multi-block metadata update is not atomic, so a crash between the writes leaves the file system in a state that is neither the old state nor the new one.
  • The classical response is fsck, the file system consistency check, which scans the metadata, cross-checks the structures, and repairs the inconsistencies. fsck is effective but slow: its cost is proportional to the file system's size, and it requires the file system to be unavailable.
  • Write-ahead logging is the protocol on which journaling is based: the log entry is written before the change it describes. The protocol has four steps — write the log entry, write the commit record, checkpoint the changes, and free the journal space — and the order of the steps is what makes it correct.
  • The journal is a circular log of transactions, each consisting of a begin record, the metadata blocks that the transaction will modify, and a commit record. The journal's records include descriptors, sequence numbers, and checksums.
  • Checkpointing applies the transaction's changes to their final locations and frees the journal space. The checkpoint pointer records how far the checkpointing has progressed, and the file system forces a checkpoint when the journal's free space falls below a threshold.
  • The journaling modes trade consistency against performance. Writeback journals metadata only and does not order the data writes (fastest, weakest). Ordered journals metadata only but writes the data before the metadata (the default, balanced). Data journals both metadata and data (slowest, strongest). Journaling provides consistency, not durability; durability requires fsync.
  • JBD2 is the journaling layer used by ext3 and ext4. It manages transactions, the journal's circular log, the checkpoint pointer, and the recovery. On ext4, the journal is inode 8 of the file system.
  • The recovery process scans the journal from the last checkpoint, identifies the committed transactions (those with valid commit records and checksums), replays them, and discards the incomplete transactions. The recovery's cost is proportional to the journal's size, not the file system's size, so it takes seconds rather than hours.
  • Log-structured file systems write all data — metadata and file contents — to a single sequential log, achieving the device's full sequential bandwidth even for random updates. Copy-on-write file systems (Btrfs, ZFS) never overwrite a block in place, so the file system's state changes only when the root pointer is updated atomically. Both mechanisms achieve crash consistency without a journal.
  • Checksums detect metadata corruption: every metadata block includes a checksum, and the file system verifies it when the block is read. Journaling provides crash consistency, and checksums provide corruption detection; the two mechanisms are complementary, and a file system needs both to be robust.

Looking ahead

This lesson presented journaling: the crash- consistency problem, the write-ahead logging protocol, the journal's structure, the journaling modes, JBD2, the recovery process, and the modern alternatives (log-structured and copy-on-write file systems). It referred several times to the modern file systems — ext4, XFS, Btrfs, ZFS — and to the features that distinguish them: extents, checksums, snapshots, copy-on-write, and RAID. Lesson 10.7 presents modern file systems: a survey of the file systems in use today, their design goals, their features, and the trade-offs they make. The lesson covers ext4, XFS, Btrfs, ZFS, and the flash-specific file systems (F2FS), and it shows how the concepts from Lessons 10.1 through 10.6 are combined in the systems that run the world's storage.

11 References

  • Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 14, Section 14.7, and Chapter 15, Section 15.6.
  • Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapter 42. Available online at ostep.org.
  • Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 4, Section 4.3.5.
  • Tweedie, S. C. Journaling the Linux ext2fs File System. USENIX Annual Technical Conference, 1998.
  • Prabhakaran, V., Arpaci-Dusseau, A. C., & Arpaci-Dusseau, R. H. Analysis and Evolution of Journaling File Systems. USENIX Annual Technical Conference, 2005.
  • Rosenblum, M., & Ousterhout, J. K. The Design and Implementation of a Log-Structured File System. ACM Transactions on Computer Systems, 1992.
  • Rodeh, O., Bacik, J., & Mason, C. BTRFS: The Linux B-Tree Filesystem. ACM Transactions on Storage, 2013.
  • McKusick, M. K., Joy, W. N., Leffler, S. J., & Fabry, R. S. A Fast File System for UNIX. ACM Transactions on Computer Systems, 1984.
  • Kerrisk, M. The Linux Programming Interface. No Starch Press, 2010. Chapter 14.
  • Love, R. Linux Kernel Development, 3rd ed., Chapter 13, The Virtual File System.
  • Bovet, D. P., & Cesati, M. Understanding the Linux Kernel, 3rd ed. O'Reilly, 2005. Chapter 12.
  • Linux man-pages: man 2 fsync, man 2 fdatasync, man 2 sync, man 5 ext4, man 8 tune2fs, man 8 e2fsck, man 1 debugfs, man 5 proc.
👤