COMP 3150Operating Systems

01 Introduction

Lesson 10.1 presented the file and directory abstractions: a file as a named, persistent sequence of bytes, and a directory as a mapping from names to files. It introduced the idea that a file has attributes — information about the file that is separate from its contents — and it noted that those attributes are stored in the file system's metadata structures, not in the file itself. This lesson examines those attributes in detail.

File metadata is the collection of information that a file system maintains about a file: everything except the file's data. The distinction is fundamental. When a program calls open, the file system reads the metadata to check permissions and to find the file's contents. When a program calls stat, the file system reads the metadata and returns it without touching the contents at all. When a file is modified, the file system updates the metadata (the modification time, the size, the block count) as well as the contents. The metadata is a first-class part of the file, and its correctness and its performance matter as much as the contents'.

The metadata is significant for three reasons. First, it is what makes a file more than a sequence of bytes: the metadata gives a file its identity (its inode number), its name (through the directory), its type, its owner, and its permissions. Second, the metadata is what the file system uses to find the file's contents: the pointers to the file's data blocks are stored in the metadata, not in the data itself. Third, the metadata is what the kernel consults on every access: the permission check, the dirty-flag check, the timestamp update, and the lookup of the file's data blocks all involve reading or writing the metadata.

This lesson presents the metadata in the order that a program encounters it: the standard stat interface (which exposes the metadata to user space), the file's identity (device and inode number, link count), the file's type and permission bits, the ownership fields, the three timestamps and their semantics, the file's size and block count (including the special case of sparse files), and the extended attributes and access control lists that modern file systems provide. It closes with a discussion of where the metadata lives — the inode, which is the subject of Lesson 10.3 — and of the cost of metadata updates, which is the motivation for journaling (Lesson 10.6) and for the design of modern file systems (Lesson 10.7).

Prerequisites

This lesson builds directly on Lesson 10.1. The file and directory abstractions, the distinction between a file's name and its identity, and the standard file operations are all assumed. The concept of an inode was introduced in Lesson 10.1 as the file's metadata structure; this lesson examines the metadata's contents, and Lesson 10.3 examines the inode's structure and lifetime. Familiarity with the Unix permission bits (introduced in section 4.11 of Lesson 10.1) is assumed, and the discussion of the permission-check algorithm in section 4.5 extends that material. No prior knowledge of the stat interface, timestamps, sparse files, extended attributes, or ACLs is required; all are introduced here. The interactive activity uses the standard Unix tools (stat, ls, touch, getfattr, getfacl), so familiarity with the Linux command line from Lesson 2.2 is helpful.

02 Learning objectives

By the end of this lesson you will be able to

  • Describe the standard metadata interface and identify the fields of the struct stat structure.
  • Explain file identity (device and inode number) and the role of the link count.
  • Describe the file type and mode bits and trace the kernel's permission-check algorithm.
  • Distinguish the three timestamps (atime, mtime, ctime) and explain when each is updated.
  • Explain sparse files, extended attributes, and access control lists, and describe where metadata lives in the file system.

This lesson contributes to U10.2 and U10.3, and to course learning outcomes CLO1, CLO5, and CLO6.

03 Required reading

  • Operating System Concepts — Chapter 13, Section 13.1, File Concept, and Chapter 14, Section 14.2, File-System Implementation. The standard treatment of file attributes and of the inode as the container of metadata.
  • OSTEP — Chapter 39, Files and Directories, and Chapter 40, File System Implementation. OSTEP's treatment of the stat interface and of the inode as the metadata structure is the clearest available account, and it is the basis for the worked example in this lesson.
  • Tanenbaum & Bos — Modern Operating Systems, Chapter 4, Sections 4.1 and 4.3, Files and File-System Implementation. The treatment of file attributes and the role of the inode.
  • Silberschatz et al. — Chapter 13, Section 13.1, and Chapter 14, Section 14.2. Read the discussion of the attributes and of the inode's fields.
  • Kerrisk, M. — The Linux Programming Interface, Chapter 15, File Attributes, and Chapter 16, Extended Attributes. The definitive reference for the stat structure, its fields, and the Linux-specific extensions.
  • Bryant, R. E., & O'Hallaron, D. R. — Computer Systems: A Programmer's Perspective, 3rd ed., Chapter 10, System-Level I/O, and Chapter 9, Virtual Memory, section 9.10. The programmer's view of file metadata and of the interaction between the file system and the page cache.
  • McKusick, M. K., Joy, W. N., Leffler, S. J., & Fabry, R. S. — A Fast File System for UNIX. ACM Transactions on Computer Systems, 1984. The paper that refined the inode structure and the metadata layout for performance, and that established many of the design principles still in use.
  • Linux man pages: man 2 stat, man 2 lstat, man 2 fstat, man 2 statx, man 2 chmod, man 2 chown, man 2 utime, man 2 utimensat, man 2 truncate, man 7 symlink, man 7 xattr, man 5 acl, man 1 stat, man 1 getfattr, man 1 setfattr, man 1 getfacl, man 1 setfacl, man 5 proc (for the /proc/<pid>/stat interface).

Reading strategy

Read the OSTEP chapters first. Their treatment of the inode as the container of file metadata, and of the stat interface as the way user space sees it, is the clearest available, and it establishes the vocabulary for the rest of the lesson. Then read the Kerrisk chapters for the definitive reference on the struct stat fields and on Linux's extended attributes. The man 2 stat page is worth reading carefully: it documents each field, its units, and the caveats that surround some of them (particularly the timestamps, which have subtle semantics). Pay particular attention to the timestamps: they are the part of the metadata whose semantics are most often misunderstood, and the kernel's policies for updating them (the relatime, noatime, and strictatime mount options) have a significant effect on performance.

04 Core content

4.1 What is file metadata?

File metadata is the collection of information that a file system maintains about a file, excluding the file's contents. It includes the file's identity, its type, its ownership, its permissions, its size, its timestamps, and any other attributes the file system supports. On Unix-like systems, the metadata is stored in the file's inode (Lesson 10.3), and it is exposed to user space through the stat family of system calls.

The metadata is not a separate entity from the file: it is the file, in the sense that the file's identity is its inode number and its contents are located through the pointers in the metadata. When you delete a file, you are removing the directory entry that refers to the metadata's inode; the metadata itself is freed when the last reference to it is gone. When you rename a file, you are changing the directory entry that refers to the inode; the metadata is unaffected. When you modify a file, you are changing the contents, and the metadata is updated to reflect the change (the modification time, the size, and the block count are all adjusted).

The metadata's layout and its relationship with the file's data are central to the file system's design. On classical Unix file systems, the metadata is stored in a fixed-size structure (the inode) whose location is determined by the file's inode number, and the file's data blocks are located through pointers stored in the inode. On modern file systems, the metadata may be stored in a more flexible structure — a B-tree, a log, or a copy-on-write tree — but the principle is the same: the metadata records the file's attributes and provides the pointers to the file's contents.

4.2 The stat interface

The metadata is exposed to user space through the stat family of system calls. The calls fill in a struct stat structure, which is the user-space view of the file's metadata. On Linux, the structure has the following fields:

struct stat {
    dev_t     st_dev;      /* device containing the file */
    ino_t     st_ino;      /* inode number (file identity) */
    mode_t    st_mode;     /* file type and permission bits */
    nlink_t   st_nlink;    /* number of hard links */
    uid_t     st_uid;      /* user ID of the owner */
    gid_t     st_gid;      /* group ID of the group */
    dev_t     st_rdev;     /* device ID (for device files) */
    off_t     st_size;     /* total size in bytes */
    blksize_t st_blksize;  /* preferred block size for I/O */
    blkcnt_t  st_blocks;   /* number of 512-byte blocks allocated */
    struct timespec st_atim;  /* time of last access */
    struct timespec st_mtim;  /* time of last modification */
    struct timespec st_ctim;  /* time of last status change */
    /* Linux also provides st_birthtim (creation time) in statx */
};

The three calls differ in how they interpret the path:

  • stat(path, buf) — fills in the structure with the metadata of the file that path refers to, following symbolic links. If path is a symbolic link, the metadata returned is that of the target, not the link.
  • lstat(path, buf) — the same, but does not follow symbolic links. If path is a symbolic link, the metadata returned is that of the link itself (its size is the length of the pathname it contains, and its type is symbolic link).
  • fstat(fd, buf) — takes an open file descriptor rather than a path, and fills in the structure with the metadata of the file the descriptor refers to. Useful for files that have been opened but whose path is not known, or that may have been renamed since they were opened.

Linux also provides statx, an extended version of stat that exposes additional fields, including the file's creation time (stx_btime), the file's attributes (stx_attributes), and the file's mount ID. The extended interface is the subject of section 4.8.

The struct stat interface and the metadata it exposes A diagram showing the struct stat structure in the middle, with its fields listed. On the left, the fields are grouped by their purpose: identity (st_dev, st_ino, st_nlink), type and permissions (st_mode), ownership (st_uid, st_gid), size (st_size, st_blksize, st_blocks), and timestamps (st_atim, st_mtim, st_ctim, st_birthtim). On the right, arrows show where each group of fields is stored in the file system's inode on disk. The diagram illustrates that the struct stat is the user-space view of the inode's contents. THE STRUCT STAT INTERFACE The user-space view of a file's metadata Purpose Identity st_dev — device st_ino — inode number st_nlink — link count Type and mode st_mode — type and permission bits Ownership st_uid — owner's UID st_gid — owner's GID Size st_size — size in bytes st_blksize — I/O block st_blocks — 512B blocks Timestamps st_atim — last access st_mtim — last modify st_ctim — last change st_birthtim — created struct stat struct stat filled in by the kernel on stat() / lstat() / fstat() dev_t st_dev ino_t st_ino mode_t st_mode nlink_t st_nlink uid_t st_uid gid_t st_gid dev_t st_rdev off_t st_size blksize_t st_blksize blkcnt_t st_blocks timespec st_atim timespec st_mtim timespec st_ctim timespec st_birthtim the kernel fills the structure from the file's inode Inode on disk inode fixed-size, ~128–256 bytes located by inode number dev, ino, nlink mode, type, perms uid, gid size, blocks atime, mtime, ctime birthtime (if supported) data block pointers direct, indirect, doubly indirect or extent pointers extended attributes named key/value pairs stored inline or externally the inode is the file's metadata; the struct stat is its user-space view The struct stat is a snapshot The kernel fills the structure with the inode's current contents at the moment of the call. The structure is a copy: changes to the inode afterward are not reflected in the copy. The three system calls differ in how they interpret the path argument.
Figure 10.2 — The struct stat interface and the metadata it exposes. The structure is the user-space view of a file's inode. Its fields group into five purposes: identity, type and mode, ownership, size, and timestamps. The kernel fills the structure from the file's inode on disk, and the structure is a snapshot of the inode's contents at the moment of the call.

4.3 File identity

A file's identity is the pair (st_dev, st_ino): the device that contains the file and the inode number within that device. The pair is necessary because an inode number is meaningful only within a single file system; the same inode number can refer to different files on different devices. Together, the pair uniquely identifies the file within the entire system.

Programs that need to determine whether two paths refer to the same file compare their (st_dev, st_ino) pairs. This is what find does to detect hard links to the same file, what rsync does to detect files that have been renamed, and what the kernel's directory-entry cache uses to recognise when two paths resolve to the same file.

The st_nlink field records the link count: the number of hard links to the file. When a hard link is created, the count is incremented; when a hard link is removed, the count is decremented. The file's inode is freed only when the count reaches zero and no process has the file open. The link count is a crucial part of the file system's bookkeeping: it is what allows the file to remain accessible through other names after one of its names is deleted, and what ensures that the file's resources are reclaimed only when the last reference is gone.

On Unix, the link count for a regular file starts at 1 when the file is created (there is one name: the file itself). For a directory, the link count starts at 2 — one for the directory's own name in its parent, and one for the . entry inside it. Each subdirectory adds one more to the count, because each subdirectory contains a .. entry that refers to its parent. This is why a directory's link count is one plus the number of its immediate subdirectories.

4.4 File type and mode bits

The st_mode field encodes two pieces of information: the file's type and its permission bits. The type is stored in the high bits of the field, and the permission bits in the low nine bits. The standard types are:

File types and their st_mode flags
Type Symbol Description
Regular file S_IFREG An ordinary file: a sequence of bytes with no special meaning to the file system.
Directory S_IFDIR A file whose contents are a sequence of directory entries.
Symbolic link S_IFLNK A file whose contents are a pathname; dereferenced on access unless lstat is used.
Character device S_IFCHR A device accessed as a stream of bytes: a terminal, a serial port, /dev/null.
Block device S_IFBLK A device accessed in fixed-size blocks: a disk, a partition.
FIFO (named pipe) S_IFIFO A named channel for inter-process communication.
Socket S_IFSOCK A Unix-domain socket bound to a pathname.

The permission bits occupy the low nine bits of st_mode and are organised into three triplets: the owner's permissions, the group's permissions, and the permissions for everyone else. Each triplet contains a read bit, a write bit, and an execute bit. The symbolic constants are S_IRUSR, S_IWUSR, S_IXUSR for the owner, S_IRGRP, S_IWGRP, S_IXGRP for the group, and S_IROTH, S_IWOTH, S_IXOTH for others.

The meaning of the execute bit depends on the file's type. For a regular file, the execute bit grants permission to execute the file as a program. For a directory, the execute bit grants permission to search the directory — that is, to look up names within it. A directory with read permission but not execute permission can be listed (the names can be read), but the entries cannot be accessed (the names cannot be resolved to inodes). A directory with execute permission but not read permission can be traversed (a path through it can be resolved) but its contents cannot be listed. The combination is unusual but legal.

Three additional bits are stored in the same field, in positions adjacent to the permission bits:

  • Set-user-ID (SUID) — When an executable file with the SUID bit is run, the process runs with the privileges of the file's owner, not the user who invoked it. This is how programs like passwd are able to modify /etc/shadow, which is writable only by root.
  • Set-group-ID (SGID) — The same, but for the group. On a directory, the SGID bit has a different meaning: files created in the directory inherit the directory's group, not the creating user's group. This is used for shared project directories.
  • Sticky bit — On a directory, the sticky bit restricts deletion: a file in a sticky directory can be deleted only by its owner, the directory's owner, or root. This is used for directories like /tmp, where many users create files but should not be able to delete each other's.

4.5 Ownership and the permission-check algorithm

The st_uid and st_gid fields record the file's owner and group. They are used for the permission check — the kernel compares the calling process's effective user and group IDs against the file's — and for accounting (which user owns how much data).

The permission check is performed on every access, and its algorithm is specific enough to be worth stating precisely. When a process tries to access a file, the kernel applies the following rules, in order:

  1. If the process's effective UID is 0 (root), the access is allowed. Root bypasses the permission checks entirely, with two important exceptions: executing a regular file requires at least one execute bit to be set, and writing to a file in a read-only file system is not allowed. The exceptions exist so that root cannot accidentally execute a non-executable file or modify a read-only file system.
  2. Otherwise, if the process's effective UID equals the file's st_uid, the owner's permission bits (the high triplet of the nine bits) are used. The group and other bits are ignored, even if they would grant more permission.
  3. Otherwise, if the process's effective GID (or one of its supplementary GIDs) equals the file's st_gid, the group's permission bits (the middle triplet) are used.
  4. Otherwise, the other permission bits (the low triplet) are used.

The order is significant. The owner check comes before the group check, and the group check comes before the other check. The kernel applies one set of bits, not all three: if the process is the owner, the owner's bits apply even if the other bits would grant more permission. This is why a file with mode ---rwxrwx (no permission for the owner, full permission for everyone else) is inaccessible to the owner, but accessible to every other user.

The check is performed against the process's effective UID and GID, not its real ones. The distinction matters for the set-user-ID and set-group-ID mechanisms: when a process runs a set-user-ID binary, its effective UID is changed to the file owner's UID, and the permission check uses the changed value. This is what allows the program to access files the invoking user could not otherwise access.

4.6 Timestamps

The metadata contains three or four timestamps, each updated under different conditions. Their semantics are subtle, and they are the part of the metadata most often misunderstood.

st_atim — access time (atime)
The time of the file's last read. Updated when a program reads the file's contents, or when a program lists a directory's contents. Not updated when the file's metadata is read (as with stat), except under the strictatime policy. The access time is useful for archival tools (which can identify files that have not been read for a long time) and for intrusion detection (which can identify files that have been read unexpectedly), but it is expensive to maintain, because it must be updated on every read.
st_mtim — modification time (mtime)
The time of the file's last content modification. Updated when a program writes to the file's contents, truncates the file, or creates a new file. Not updated when the file's metadata is changed (changing the permission bits or the owner does not update the mtime). The modification time is the timestamp that most tools use as the file's "version" — it is what make compares to decide whether a target needs to be rebuilt, and what rsync compares to decide whether a file has changed.
st_ctim — status change time (ctime)
The time of the file's last metadata change. Updated when the inode is modified: when the permission bits are changed, when the owner is changed, when the link count is changed, when the file's size changes, when the file is renamed, and when the file's content is modified (because content modification also updates the inode's size and modification time). The ctime is therefore a superset of the mtime: every mtime update also updates the ctime, but the ctime can be updated without the mtime (a permission change updates the ctime alone).
st_birthtim — creation time (birthtime)
The time at which the file was created. Not all file systems support it, and not all kernels expose it: classical Unix file systems did not track the creation time, and the classical struct stat has no field for it. Modern file systems (ext4, XFS, Btrfs, ZFS, APFS, NTFS) do track the creation time, and Linux exposes it through the extended statx interface as stx_btime. The field is sometimes called the birthtime to distinguish it from the ctime.

The three classical timestamps — atime, mtime, and ctime — are the ones that every Unix system provides. The birthtime is a modern addition. The distinctions between them are easy to confuse, and it is worth restating: mtime is the content, ctime is the metadata, atime is the read.

The three timestamps: atime, mtime, and ctime A timeline showing the life of a file with four events marked. At time T0 the file is created: all three timestamps are set to T0. At time T1 the file is read: only atime is updated. At time T2 the file is written: mtime and ctime are both updated. At time T3 the permission bits are changed: only ctime is updated. The diagram shows three horizontal tracks, one for each timestamp, with the value of the timestamp at each point in time. The visualisation makes clear that atime tracks reads, mtime tracks content changes, and ctime tracks any metadata change (including content changes). THE THREE TIMESTAMPS time T0 create T1 read T2 write T3 chmod atime last read T0 T1 mtime last write T0 T2 ctime last metadata change T0 T2 T3 birthtime creation (if supported) T0 — set once at creation; never changes The pattern atime tracks reads · mtime tracks content changes · ctime tracks any inode change (including content) Every write updates both mtime and ctime; a chmod or chown updates only ctime; a read updates only atime.
Figure 10.2b — The three timestamps. The timeline shows the life of a file with four events: creation (T0), a read (T1), a write (T2), and a permission change (T3). The three tracks show how each timestamp evolves. The pattern is that atime tracks reads, mtime tracks content changes, and ctime tracks any metadata change — including content changes, which are metadata changes as well. The birthtime is set once at creation and never changes.

The timestamps are also the subject of a Linux-specific optimisation that is worth understanding. Updating atime on every read is expensive, because it turns every read into a write to the inode. The write is small — a few bytes — but it requires the inode to be located, the disk block to be read, modified, and written back, and it can be a significant fraction of the cost of a read that would otherwise be served entirely from the page cache. To avoid this cost, Linux provides three mount options that control when atime is updated:

  • strictatime — Update atime on every read. This is the classical behaviour, and it is what the POSIX standard requires. It is rarely used because of its cost.
  • relatime — Update atime only if the file's current atime is older than its mtime or ctime, or if more than a day has passed since the last update. This is the default on most Linux systems. It preserves the timestamps' usefulness for most purposes while eliminating the cost of updating them on every read.
  • noatime — Never update atime. This is the fastest option, but it makes atime useless. It is used on systems where performance matters more than the access-time information, and on systems where the atime field is not needed (many modern systems use noatime on their root file systems).

The choice of policy is a trade-off between performance and the usefulness of the atime field. Most modern systems choose relatime as a compromise: it updates the field often enough for archival tools and intrusion detection, but rarely enough to avoid the cost on the common case of a file that is read repeatedly.

4.7 Size and block counts

The metadata records two measures of the file's size: st_size, the logical size in bytes, and st_blocks, the number of 512-byte blocks actually allocated on disk. The two are usually related (the block count is at least the logical size divided by 512), but they can differ in both directions.

The most common reason for a difference is block alignment. A file system allocates space in fixed-size blocks (typically 4 KB on ext4, 8 KB or larger on some others), and the last block of a file may be only partially used. A file with a logical size of 1 byte occupies one full block on disk, so st_size is 1 and st_blocks is 8 (one 4 KB block divided into 512-byte units). The difference is the file system's internal fragmentation, and it is the price of the file system's block granularity.

The other reason is the opposite: a file's st_size can be much larger than the space it actually occupies. This is the case for a sparse file, a file whose logical size includes "holes" — regions that contain no data. A sparse file is created when a program seeks past the end of the file and writes data there, or when a program explicitly creates a sparse file with ftruncate or lseek. The file system does not allocate blocks for the holes: the metadata's block pointers are NULL for the hole's regions, and the file system returns zeros when a program reads them. The result is a file whose st_size may be many gigabytes but whose st_blocks may be only a few megabytes.

Sparse files are useful for several purposes. A virtual machine's disk image is typically sparse: the image's logical size is the size of the virtual disk, but the space actually allocated is only the space the guest operating system has written. A database's file might be sparse: the file is pre-allocated to a maximum size, but the space for unused records is not allocated. A scientific computation's output file might be sparse: the file is large, but only the regions the computation actually produced contain data.

The handling of sparse files is one of the differences between file systems. The classical Unix file system and its descendants (ext2, ext3, ext4) handle sparse files by leaving the block pointers for a hole empty. Modern file systems (XFS, Btrfs, ZFS) handle them more efficiently, using extents — a range of contiguous blocks with a single descriptor — which allows a hole to be represented by a single descriptor rather than by the absence of many pointers. And some file systems support hole punching: an operation that creates a hole in an existing file, freeing the blocks that were allocated there. Hole punching is used in files that have been written and then have regions marked as "do not care", such as the tail of a virtual machine's disk image after a snapshot.

4.8 Extended attributes

The classical struct stat exposes the metadata that the POSIX standard specifies, but modern file systems support additional metadata that does not fit in the classical structure. This additional metadata is stored in extended attributes (xattrs), which are named key-value pairs associated with a file.

An extended attribute has a name and a value. The name is a string in a namespaced format: on Linux, the namespaces are user. (for user-defined attributes), trusted. (for attributes visible only to privileged processes), security. (for security attributes such as SELinux labels and capabilities), and system. (for system attributes such as POSIX ACLs). The value is a byte string whose meaning is determined by the namespace and by the application that uses it.

Extended attributes are used for several purposes.

  • Access control lists. On Linux, POSIX ACLs are stored as extended attributes in the system. namespace. The classical permission bits remain in the inode; the ACL extends them with additional entries.
  • Security labels. SELinux and AppArmor store their security context in security. attributes.
  • File capabilities. The security.capability attribute stores the capabilities that a program should have when it is executed, as an alternative to the set-user-ID mechanism.
  • File system features. A file system may use extended attributes for its own bookkeeping: for example, to record the file's checksums, its encryption keys, or its compression flags.
  • Application metadata. An application may store arbitrary metadata in the user. namespace: the source URL of a downloaded file, the encoding of a document, the thumbnail of an image.

The support for extended attributes varies between file systems. On Linux, ext4 supports extended attributes up to a limited total size (typically a few kilobytes per inode), and they are stored either inline in the inode or in a separate block pointed to by the inode. XFS, Btrfs, and ZFS support extended attributes with larger limits. The interface for manipulating them is setxattr, getxattr, listxattr, and removexattr, and the command-line tools are setfattr and getfattr.

4.9 Access control lists

The classical Unix permission model is a good fit for the majority of cases: a file has an owner, a group, and a set of permissions, and that is sufficient. But it has a fundamental limitation: it can express permissions for at most one group. A file that needs to be readable by three different groups, or by a specific list of users, cannot be expressed in the classical model without creating additional groups or duplicating the file.

Access control lists (ACLs) extend the model with a list of entries, each specifying a user or group and a set of permissions. The list is consulted after the classical permission bits: if the classical bits already allow the access, the ACL is not consulted; if they do not, the ACL is checked for an entry that grants the access.

A POSIX ACL has up to three kinds of entry:

  • User entries — one for the file's owner (the user:: entry, which corresponds to the classical owner bits) and zero or more for specific users (the user:alice: entries).
  • Group entries — one for the file's owning group (the group:: entry, which corresponds to the classical group bits) and zero or more for specific groups (the group:staff: entries).
  • Other entries — one for everyone else (the other:: entry, which corresponds to the classical other bits).
  • Mask entry — a special entry that limits the permissions granted by the user and group entries. The mask is intersected with the permissions in each entry, so the effective permission of a user entry is the entry's permission AND the mask.

The ACL's entries are stored in an extended attribute (system.posix_acl_access), and the classical permission bits continue to exist alongside them. The classical bits and the ACL are kept in sync: changing the classical bits with chmod updates the ACL's owner, group, and other entries, and changing those entries in the ACL updates the classical bits. This is why chmod works on a file with an ACL, and why the interaction between the two is more subtle than it first appears.

ACLs are a powerful tool, but they are also a source of complexity. A file with an ACL behaves differently from a file without one: the permission check requires reading the ACL, the ls -l output shows a + after the mode to indicate the presence of an ACL, and the classical permission bits have a different meaning (they are the maximum, not the actual permissions). Most systems therefore use ACLs sparingly, for the cases where the classical model is genuinely insufficient.

4.10 Where metadata lives

The metadata is stored in the file system's own structures, separate from the file's data. On classical Unix file systems, this structure is the inode, a fixed-size record whose location is determined by the file's inode number. The inode contains all the file's metadata — its type, its permissions, its owner, its size, its timestamps, and the pointers to its data blocks — and it is read and written as a unit.

Modern file systems have relaxed the fixed-size, fixed-location assumption of the classical inode. Ext4 still uses fixed-size inodes but supports extended attributes that spill into additional blocks. XFS uses variable-size inodes that can grow as needed. Btrfs and ZFS store metadata in B-trees and copy-on-write trees, with no separate inode structure at all: the file's metadata is a node in a tree, and the tree is updated atomically. The struct stat interface is preserved across these changes — the metadata is still exposed to user space in the same way — but the internal representation is different.

The separation of metadata from data has a cost. Reading a file requires the file system to read the file's metadata (to find the data blocks) and then read the data blocks themselves. For a small file, the metadata read dominates; for a large file, the data reads dominate. And writing a file requires updating both the data blocks and the metadata (the size, the modification time, the block pointers), so a single write can involve several disk accesses to different parts of the file system. This is the subject of Lesson 10.5 (file allocation) and Lesson 10.6 (journaling), which examine the strategies for laying out the metadata and for keeping it consistent in the face of crashes.

4.11 Common misconceptions

"Metadata is optional."

Metadata is not optional: it is what makes a file a file. Without the metadata, the file has no identity, no name, no owner, no permissions, and no way to locate its contents. Every file has metadata, and the metadata is as much a part of the file as its contents.

"st_size is the space the file occupies on disk."

st_size is the logical size in bytes; st_blocks × 512 is the space actually allocated. The two can differ: a file with a logical size of 1 byte occupies at least one block, and a sparse file can have a logical size much larger than its allocated space.

"mtime and ctime are the same."

They are not. mtime is the time of the last content modification; ctime is the time of the last metadata change. Every mtime update also updates ctime, but not vice versa: a chmod or chown updates ctime alone.

"The owner of a file always has all permissions."

The owner is subject to the owner's permission bits, which can be set to deny any permission. A file with mode ---rwxrwx is inaccessible to its owner but accessible to everyone else.

"Root can do anything."

Root bypasses the permission checks, but not unconditionally: root cannot execute a file with no execute bits set, and root cannot write to a read-only file system. The exceptions exist for safety.

"A hard link and a copy are the same."

A copy is a new file with its own metadata and its own data blocks; modifying the copy does not affect the original. A hard link is the same file, reached through a different name; modifying the file through either name affects the other. The two have different link counts and different inode numbers.

"Extended attributes are part of the file's contents."

Extended attributes are metadata, not contents. They are stored in the inode (or in blocks pointed to by it), they are read with a different interface, and they are not part of the byte stream that read and write see.

"The struct stat is the file's metadata."

The struct stat is the user-space view of the metadata; the metadata itself lives in the inode (or its equivalent). The structure is a copy, filled in by the kernel at the moment of the call, and changes to the file's metadata afterward are not reflected in the copy.

05 Worked example

The following example interprets the metadata of several files, traces the permission-check algorithm through a specific case, and demonstrates the behaviour of sparse files.

Interpreting the metadata of a file

Suppose a program calls stat on a file and receives the following structure (fields abbreviated):

st_dev    = 0x0801            /* device 8, minor 1 */
st_ino    = 262147
st_mode   = 0x81A4            /* 0100644 in octal */
st_nlink  = 3
st_uid    = 1000              /* alice */
st_gid    = 100              /* staff */
st_rdev   = 0
st_size   = 40960             /* 40 KB */
st_blksize = 4096
st_blocks = 80                /* 80 × 512 = 40 KB */
st_atim   = 1712345678.123456789
st_mtim   = 1712345000.987654321
st_ctim   = 1712345000.987654321

The fields can be interpreted as follows.

  • Identity. (st_dev, st_ino) = (0x0801, 262147) uniquely identifies the file within the system. st_nlink = 3 means there are three hard links to the file, so the file has three names.
  • Type and mode. The mode 0x81A4 is 0100644 in octal. The 010 prefix is S_IFREG, so the file is a regular file. The low three octal digits are 644, so the owner has read and write, and the group and others have read only.
  • Ownership. The file is owned by user ID 1000 (alice) and group ID 100 (staff).
  • Size. The logical size is 40 KB, and the allocated size is 80 blocks × 512 bytes = 40 KB. The two match, so the file has no holes and occupies exactly as much space on disk as its logical size suggests. (Actually, 40 KB is a multiple of 4 KB, so no partial block is involved; a file with a size of 40 KB + 1 byte would have the same st_blocks but a different st_size.)
  • Timestamps. The access time is about 11 minutes later than the modification time, which means the file was read after it was written. The modification and change times are identical, which means the last operation on the file was a content modification (not a metadata change alone). If a chmod had been run after the write, the ctim would be later than the mtim.

Tracing the permission check

A process running as user ID 2000 (bob) tries to open the file with O_WRONLY. Bob is not the owner (UID 1000), but he is a member of group 100 (staff). The kernel's permission check proceeds as follows:

  1. Is the process's effective UID 0? No (bob's UID is 2000, not 0). Root's bypass does not apply.
  2. Is the process's effective UID equal to the file's st_uid? No (2000 ≠ 1000). The owner's permission bits are not used.
  3. Is the process's effective GID (or any supplementary GID) equal to the file's st_gid? Yes (bob is a member of group 100). The group's permission bits are used.
  4. Do the group's permission bits include the requested access? The mode is 644, so the group's bits are r--: read permission only, no write permission. The requested access is write, so the check fails.

The open call returns -1 with errno set to EACCES. Bob cannot write to the file, even though he can read it.

Now suppose the file's mode is changed to 664 (owner and group read-write, others read). The owner's bits are still rw-, the group's bits become rw-, and the others' bits stay r--. Bob's check now reaches step 4 with the group's bits rw-, which include the requested write access, so the check succeeds. The open call returns a file descriptor.

Finally, suppose the file is changed to mode 600 (owner read-write only). Bob's check reaches step 3 with the group's bits ---, which do not include the requested write access, so the check fails. Bob cannot write to the file, even though he is a member of the file's group. The group membership is not sufficient; the group's bits must also grant the access.

Demonstrating sparse files

A program creates a file, seeks past the end, and writes a byte:

#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>

int main(void) {
    int fd = open("sparse.bin", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (fd < 0) { perror("open"); return 1; }

    /* write one byte at offset 0 */
    if (write(fd, "A", 1) != 1) { perror("write"); return 1; }

    /* seek to 1 GB past the start */
    if (lseek(fd, 1024L * 1024 * 1024, SEEK_SET) == -1) {
        perror("lseek"); return 1;
    }

    /* write one byte at the new position */
    if (write(fd, "B", 1) != 1) { perror("write"); return 1; }

    close(fd);
    return 0;
}

After the program runs, the file's metadata is examined:

$ stat sparse.bin
  File: sparse.bin
  Size: 1073741825  Blocks: 8  IO Block: 4096  regular file
Device: 801h/2049d    Inode: 524291      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   alice)   Gid: ( 100/  staff)
Access: 2025-04-05 10:15:32.123456789 +0000
Modify: 2025-04-05 10:15:32.123456789 +0000
Change: 2025-04-05 10:15:32.123456789 +0000
 Birth: 2025-04-05 10:15:32.123456789 +0000

The file's logical size is 1073741825 bytes — one byte more than 1 GB. Its allocated size is 8 blocks of 512 bytes, or 4 KB. The difference is the hole: the file system has allocated only the two blocks that contain the two bytes written (one block for the first byte, one block for the last byte), and the 1 GB between them is a hole. The hole contains no data, and reading from it returns zeros without any disk access.

The block count of 8 (one 4 KB block) is the minimum allocation for a file with data at two distant offsets: the file system allocates a block for the first byte and a block for the last byte, and the two blocks happen to fall in the same 4 KB unit — no, that cannot be, since the offsets are 1 GB apart. The block count of 8 means the file system has allocated 4 KB of disk for the file's two blocks. This illustrates the difference between the logical size and the physical size: the file appears to be 1 GB, but it occupies 4 KB on disk.

If the file is copied with a program that reads and writes every byte — as cp does without the --sparse option — the copy loses the sparseness, because the program reads a stream of bytes (mostly zeros) and writes them all, allocating blocks for the entire 1 GB. If the file is copied with a program that is aware of holes — as cp --sparse or rsync --sparse is — the copy preserves the sparseness by seeking past the holes instead of writing zeros for them.

Comparing stat, lstat, and fstat

Suppose a symbolic link link.txt points to a regular file target.txt with a size of 1000 bytes. The three calls produce different results:

Call Argument st_size st_mode type
stat("link.txt", &buf) The link's path 1000 Regular file (the target)
lstat("link.txt", &buf) The link's path 8 Symbolic link (the link itself)
fstat(fd, &buf) An open FD for the target 1000 Regular file (the target)

The stat call follows the link, so it returns the target's metadata: size 1000, type regular file. The lstat call does not follow the link, so it returns the link's metadata: size 8 (the length of the pathname the link contains, assuming the target's path is 8 characters long), type symbolic link. The fstat call operates on the open file descriptor, which was obtained by opening the target (the open call follows the link), so it returns the target's metadata.

The distinction matters for programs that need to distinguish between a symbolic link and its target: a backup program that wants to preserve symbolic links rather than copy their targets, a security tool that wants to check whether a path is a symbolic link before following it, and a shell that implements the -L and -h test operators for file existence.

06 Interactive activity

Activity · explore file metadata on a live system

The goal of this activity is to use the Linux command line to explore file metadata, to observe the timestamps' behaviour, and to demonstrate the permission-check algorithm and the effects of sparse files. Work through the steps in your course VM.

  1. Inspect file metadata. Use the stat command to see a file's metadata, and identify each field.
    stat /etc/passwd
    stat /bin/ls
    stat /tmp
    For each file, identify: its inode number, its device, its type, its mode, its owner and group, its size, its block count, and its three timestamps. Note the difference between the logical size (Size) and the allocated size (Blocks × 512).
  2. Observe the timestamps. Create a file and observe how its timestamps change as you perform different operations.
    # Create a file
    echo "hello" > /tmp/timestamps.txt
    stat /tmp/timestamps.txt
    
    # Read the file
    cat /tmp/timestamps.txt
    sleep 2
    stat /tmp/timestamps.txt     # atime should have changed
    
    # Write to the file
    echo "world" >> /tmp/timestamps.txt
    sleep 2
    stat /tmp/timestamps.txt     # mtime and ctime should have changed
    
    # Change the permissions
    chmod 600 /tmp/timestamps.txt
    sleep 2
    stat /tmp/timestamps.txt     # only ctime should have changed
    After each step, observe which timestamps changed. The pattern should be: a read updates only Access; a write updates both Modify and Change; a chmod updates only Change.
  3. Examine the mount options for atime. Check the mount options for the file system that holds your test file.
    findmnt -T /tmp -o TARGET,SOURCE,FSTYPE,OPTIONS
    cat /proc/self/mounts | grep -E 'ext4|xfs|btrfs'
    Look for the options relatime, noatime, or strictatime. Which is in effect? If the file system is mounted with relatime (the common default), the atime may not update on every read: it updates only when the current atime is older than the mtime or ctime, or when more than a day has passed. Observe whether your test in the previous step actually updated the atime.
  4. Test the permission-check algorithm. Create a file and set its mode to various values, and observe the effect on a second user (if you have sudo and access to a second user such as nobody).
    echo "secret" > /tmp/perms.txt
    chmod 640 /tmp/perms.txt
    ls -l /tmp/perms.txt
    
    # Attempt to read as another user
    sudo -u nobody cat /tmp/perms.txt 2>&1
    
    # Change to 644 — the group's read permission becomes world's
    chmod 644 /tmp/perms.txt
    sudo -u nobody cat /tmp/perms.txt
    
    # Change to 600 — only the owner can read
    chmod 600 /tmp/perms.txt
    sudo -u nobody cat /tmp/perms.txt 2>&1
    
    # Change to 604 — everyone except the group can read
    chmod 604 /tmp/perms.txt
    sudo -u nobody cat /tmp/perms.txt
    At each step, identify which set of permission bits the nobody user matches, and whether the check succeeds. The nobody user is neither the owner nor (usually) a member of the file's group, so the "other" bits are used.
  5. Demonstrate sparse files. Write a small program that creates a sparse file, and compare its logical and physical sizes.
    #include <fcntl.h>
    #include <stdio.h>
    #include <unistd.h>
    
    int main(void) {
        int fd = open("sparse.bin", O_WRONLY | O_CREAT | O_TRUNC, 0644);
        if (fd < 0) { perror("open"); return 1; }
    
        write(fd, "A", 1);
        lseek(fd, 1024L * 1024 * 1024, SEEK_SET);
        write(fd, "B", 1);
        close(fd);
        return 0;
    }
    gcc -o sparse sparse.c
    ./sparse
    stat sparse.bin
    du -h sparse.bin       # disk usage (allocated)
    ls -lh sparse.bin      # logical size
    Compare the three outputs. The logical size is 1 GB + 1 byte; the disk usage should be a few kilobytes. Then copy the file with cp and with cp --sparse=always, and compare the copies' disk usage:
    cp sparse.bin dense.bin
    cp --sparse=always sparse.bin sparse-copy.bin
    du -h sparse.bin dense.bin sparse-copy.bin
    The dense copy loses the sparseness; the sparse copy preserves it.
  6. Inspect extended attributes. On Linux, the getfattr and setfattr commands manipulate extended attributes.
    # Create a file
    echo "hello" > /tmp/xattr-demo.txt
    
    # List its extended attributes
    getfattr -d /tmp/xattr-demo.txt
    
    # Set a user attribute
    setfattr -n user.comment -v "this is a demo" /tmp/xattr-demo.txt
    getfattr -d /tmp/xattr-demo.txt
    
    # Read the attribute
    getfattr -n user.comment /tmp/xattr-demo.txt
    
    # Remove the attribute
    setfattr -x user.comment /tmp/xattr-demo.txt
    Note that the user. namespace is readable and writable by the file's owner; the other namespaces require special privileges. Also check whether the file system supports extended attributes at all: if setfattr fails with EOPNOTSUPP, the file system does not.
  7. Inspect access control lists. On Linux, the getfacl and setfacl commands manipulate ACLs.
    # Create a file
    echo "shared" > /tmp/acl-demo.txt
    
    # Show the ACL
    getfacl /tmp/acl-demo.txt
    
    # Add an entry for a specific user
    sudo setfacl -m u:nobody:r /tmp/acl-demo.txt
    getfacl /tmp/acl-demo.txt
    ls -l /tmp/acl-demo.txt   # note the + after the mode
    
    # Attempt to read as nobody
    sudo -u nobody cat /tmp/acl-demo.txt
    
    # Remove the ACL entry
    sudo setfacl -x u:nobody /tmp/acl-demo.txt
    getfacl /tmp/acl-demo.txt
    After adding the ACL entry, the ls -l output shows a + after the mode, which indicates that the file has an ACL. The getfacl output shows the ACL's entries, including the mask entry. Observe how the classical permission bits interact with the ACL entries.
  8. Observe the metadata of a sparse file system operation. The metadata is updated when a file's size changes. Create a file, write to it, and observe the block count grow.
    echo "a" > /tmp/grow.txt
    stat -c 'size=%s blocks=%b' /tmp/grow.txt
    
    # Append data
    dd if=/dev/zero bs=1M count=10 >> /tmp/grow.txt
    stat -c 'size=%s blocks=%b' /tmp/grow.txt
    
    # Truncate
    truncate -s 1 /tmp/grow.txt
    stat -c 'size=%s blocks=%b' /tmp/grow.txt
    After the append, the logical size increases by 10 MB, and the block count increases accordingly. After the truncate, the logical size drops to 1 byte, and the block count drops as well — the blocks that held the file's contents are freed.
  9. Read the man pages. Read the man pages that document the metadata interface.
    man 2 stat
    man 2 statx
    man 5 proc | grep -A 20 '/proc/\[pid\]/stat'
    man 1 stat
    For each, write a sentence describing its main contribution to your understanding of file metadata. The man 2 stat page documents each field of the structure; the man 2 statx page documents the extended interface; the /proc/<pid>/stat interface is a related structure (for processes, not files) that uses the same conventions.

Reflect: the metadata is a surprisingly rich source of information about a file. It records not only what the file is but also how it has been used: when it was created, when it was last modified, when it was last read, and whether it is being read more or less often than it is written. This information is used by a wide range of tools: backup programs use the mtime to decide what to back up, archival tools use the atime to decide what to migrate, intrusion-detection tools use the ctime to detect unauthorized modifications, and the kernel itself uses the metadata to enforce permissions and to find the file's data. Which of these uses do you think is the most important, and which do you think is the most frequently overlooked? What would you do differently if you were designing the metadata interface today?

07 Knowledge check

Check your understanding

Question 1

Define file metadata and identify the fields of the struct stat structure. What is the relationship between the structure and the file's inode?

Show answer
File metadata is the collection of information that a file system maintains about a file, excluding the file's contents: its identity, type, ownership, permissions, size, timestamps, and any other attributes the file system supports. The struct stat structure is the user-space view of the metadata, filled in by the kernel when a program calls stat, lstat, or fstat. Its fields are:
  • st_dev — the device containing the file
  • st_ino — the inode number (the file's identity within the device)
  • st_mode — the file's type and permission bits
  • st_nlink — the number of hard links to the file
  • st_uid — the user ID of the owner
  • st_gid — the group ID of the group
  • st_rdev — the device ID, for device files
  • st_size — the logical size of the file in bytes
  • st_blksize — the preferred block size for I/O
  • st_blocks — the number of 512-byte blocks allocated
  • st_atim — the time of last access
  • st_mtim — the time of last content modification
  • st_ctim — the time of last metadata change
  • st_birthtim — the creation time, available on modern file systems through statx
The relationship between the structure and the inode is that the structure is a copy of the inode's contents at the moment of the call. The kernel reads the inode (which is the file's metadata structure on disk) and fills in the user-space structure with the fields it exposes. The structure is a snapshot: changes to the inode after the call are not reflected in the structure. The three calls — stat, lstat, and fstat — differ in how they interpret the argument: stat follows symbolic links, lstat does not, and fstat operates on an open file descriptor rather than a path.

Question 2

What is a file's identity, and why is the pair (st_dev, st_ino) used rather than just st_ino? What is the link count, and how is it maintained?

Show answer
A file's identity is the pair (st_dev, st_ino): the device that contains the file and the inode number within that device. The pair is necessary because an inode number is meaningful only within a single file system: the same inode number can refer to different files on different devices, and the same file cannot be uniquely identified by its inode number alone if the system has multiple file systems. The pair uniquely identifies the file within the entire system, and it is what programs compare when they need to determine whether two paths refer to the same file. The link count (st_nlink) is the number of hard links to the file: the number of directory entries that refer to the file's inode. It is incremented when a hard link is created (with the link system call, or with the ln command without the -s flag) and decremented when a hard link is removed (with the unlink system call, or with rm). The file's inode is freed only when the link count reaches zero and no process has the file open. This is what allows the file to remain accessible through other names after one of its names is deleted, and what ensures that the file's resources are reclaimed only when the last reference is gone. On Unix, a regular file's link count starts at 1 when the file is created; a directory's link count starts at 2 and grows by one with each subdirectory, because each subdirectory contains a .. entry that refers to its parent.

Question 3

Describe the kernel's permission-check algorithm for file access. What is the order of the checks, and why does the order matter? What are the exceptions for the root user?

Show answer
The kernel's permission-check algorithm proceeds in four steps, in order.
  1. If the process's effective UID is 0 (root), the access is allowed, with two exceptions: executing a regular file requires at least one execute bit to be set, and writing to a file in a read-only file system is not allowed.
  2. Otherwise, if the process's effective UID equals the file's st_uid, the owner's permission bits (the high triplet) are used. The group and other bits are ignored, even if they would grant more permission.
  3. Otherwise, if the process's effective GID (or one of its supplementary GIDs) equals the file's st_gid, the group's permission bits (the middle triplet) are used.
  4. Otherwise, the other permission bits (the low triplet) are used.
The order matters because the kernel applies one set of bits, not all three. The first matching category determines the check: if the process is the owner, the owner's bits apply even if the other bits would grant more permission. This is why a file with mode ---rwxrwx is inaccessible to its owner (the owner's bits deny access, and the group and other bits are not consulted) but accessible to everyone else. The exceptions for root exist for safety: root cannot accidentally execute a file with no execute bits set, and root cannot write to a read-only file system. The check is performed against the process's effective UID and GID, not its real ones. The distinction matters for the set-user-ID and set-group-ID mechanisms, which change the effective IDs so that a program can run with the privileges of the file's owner.

Question 4

Distinguish the three timestamps (atime, mtime, ctime). For each, state what it records and when it is updated. Give an example of an operation that updates ctime but not mtime.

Show answer
The three timestamps differ in what they record and when they are updated.
  • atime (access time) records the time of the file's last read. It is updated when a program reads the file's contents, or lists a directory's contents. It is not updated when the file's metadata is read (as with stat), except under the strictatime mount option.
  • mtime (modification time) records the time of the file's last content modification. It is updated when a program writes to the file's contents, truncates the file, or creates a new file. It is not updated when only the file's metadata changes.
  • ctime (status change time) records the time of the file's last metadata change. It is updated when the inode is modified: when the permission bits are changed, when the owner is changed, when the link count is changed, when the file's size changes, when the file is renamed, and when the file's content is modified. The ctime is therefore a superset of the mtime: every mtime update also updates the ctime, but the ctime can be updated without the mtime.
An example of an operation that updates ctime but not mtime is chmod: changing a file's permission bits modifies the inode (the mode field) but does not touch the file's contents, so the ctime is updated and the mtime is not. The same is true for chown (changing the owner), chgrp (changing the group), and link (adding a hard link, which updates the link count). The pattern is that operations that change the file's metadata but not its contents update only the ctime.

Question 5

What is a sparse file? How does its logical size differ from its allocated size, and how does the file system represent the holes? Give two examples of applications in which sparse files are useful.

Show answer
A sparse file is a file whose logical size includes regions — called holes — that contain no data. The file system does not allocate blocks for the holes; the metadata's block pointers for the hole's regions are empty, and a read from a hole returns zeros. The file's logical size (st_size) includes the holes and can be very large; its allocated size (st_blocks × 512) includes only the blocks that actually hold data and can be very small. For example, a file with a logical size of 1 GB and two bytes of data at the start and the end can have an allocated size of only a few kilobytes, because the 1 GB of zeros between the two bytes is a hole. Sparse files are created when a program seeks past the end of a file and writes data there (as with lseek followed by write), or when a program explicitly creates a hole with ftruncate or fallocate. They are preserved by programs that are aware of holes (such as cp --sparse=always and rsync --sparse) and lost by programs that read and write every byte (such as cp without the option). Two examples of applications: virtual machine disk images, where the image's logical size is the size of the virtual disk but only the space the guest has actually written is allocated — this allows a VM to have a large disk without consuming the full space, and it allows snapshots to be stored efficiently; and scientific and database files, where a file is pre-allocated to a maximum size but most of the space is not yet used — the file system allocates blocks only as the data grows, and the file's logical size is the maximum while its allocated size is the actual usage. Both cases benefit from the same property: the file appears to be much larger than the space it actually consumes, which allows programs to use simple, fixed-size file layouts without paying the cost of allocating the full space upfront.

08 Unit discussion

Discussion prompt

File metadata is one of those things that nobody thinks about until it goes wrong. When a file's mtime is wrong, make rebuilds things it should not; when a file's ctime is wrong, an intrusion detection system misses a compromise; when a file's permission bits are wrong, a program either cannot access a file it needs or grants access to a file it should protect. The metadata is silent, but its correctness matters.

This lesson has presented the classical metadata interface: the struct stat structure, its fields, and the semantics of each. But the interface is not the whole story. Modern file systems extend the metadata in many directions: extended attributes allow arbitrary key-value pairs to be attached to a file; ACLs extend the classical permission model with per-user and per-group entries; the statx interface adds new fields (birthtime, mount ID, file attributes); and the trend is for each new file system to add its own metadata. The result is a metadata landscape that is more complex and less portable than the classical interface suggests.

Is this a problem? Should file metadata be more standardised, so that a program that reads metadata on one system can interpret it on another? Or is the diversity appropriate, since different file systems have different capabilities and different workloads need different metadata? Consider the cloud storage services that have displaced traditional file systems for many workloads: they typically have a much smaller metadata model (a key, a value, and a few attributes), and they sacrifice much of the classical model's richness for scalability and simplicity. Which model is the right one for the next decade?

Post your response to the Unit 10 discussion forum and reply to at least two classmates. Supports CLO1, CLO5, and CLO6.

09 Practice exercise

This ungraded exercise consolidates the lesson and prepares you for Lesson 10.3. Aim to spend 60–90 minutes on it.

  1. Interpret a struct stat. For the following file metadata, interpret each field, identify the file's type, compute its access permissions, and state whether the file has holes.
    st_dev    = 0x0801
    st_ino    = 524291
    st_mode   = 0x81A4
    st_nlink  = 1
    st_uid    = 1000
    st_gid    = 100
    st_size   = 1073741825
    st_blocks = 8
    Then state the file's logical size and its allocated size in bytes, and explain the difference.
  2. Compare stat, lstat, and fstat. Create a symbolic link to a file and compare the metadata returned by the three calls. What is the link's st_size, and what does it represent? What is the target's st_size? Write the answers in your own words.
  3. Trace the permission check. A file has the following metadata: st_uid = 1000, st_gid = 100, st_mode = 0640. For each of the following processes, trace the permission check and state whether the process can read the file, write to the file, or execute the file:
    • Process runs as UID 1000, GID 100.
    • Process runs as UID 1000, GID 200 (not a member of 100).
    • Process runs as UID 2000, GID 100.
    • Process runs as UID 2000, GID 200.
    • Process runs as UID 0, GID 0.
    For each case, state which permission bits apply and why.
  4. Observe the timestamps. Create a file, then perform a sequence of operations: write to it, read from it, change its permissions, change its owner (if you have permission), and rename it. After each operation, run stat and record which timestamps changed. Present the results in a table and explain each change in terms of what the operation does to the file's contents and metadata.
  5. Compute the disk usage of sparse files. Create a sparse file of 1 GB with two bytes of data (one at the start and one at the end), then compute its logical size and its allocated size. Copy it with cp, cp --sparse=always, and cp --sparse=never, and compare the disk usage of the three copies. Explain the differences.
    du -h --apparent-size sparse.bin   # logical size
    du -h sparse.bin                    # allocated size
    ls -lh sparse.bin                   # human-readable logical size
    stat -c 'size=%s blocks=%b' sparse.bin
  6. Compare the metadata of files on different file systems. If your system has multiple file systems, create files on each and compare their metadata. Do all file systems support st_birthtim? Do they support extended attributes? Do they support ACLs? The answers depend on the file system's features.
    # Find the file systems
    df -T
    
    # Create a file on each and check its metadata
    for dir in /tmp /var/tmp /home; do
        echo "=== $dir ==="
        touch $dir/test-$$.txt
        stat $dir/test-$$.txt | head -8
        rm $dir/test-$$.txt
    done
  7. Write a program that reads metadata. Write a C program that takes a path as its argument, calls lstat, and prints the file's type, size, mode (in symbolic form), owner, group, link count, and three timestamps in human-readable form. Use the symbolic constants (S_ISREG, S_ISDIR, S_ISLNK, and so on) to determine the type, and use strftime to format the timestamps.
    #include <sys/stat.h>
    #include <stdio.h>
    #include <time.h>
    
    int main(int argc, char **argv) {
        struct stat st;
        if (lstat(argv[1], &st) == -1) {
            perror("lstat");
            return 1;
        }
    
        /* determine the type */
        if (S_ISREG(st.st_mode))       printf("regular file\n");
        else if (S_ISDIR(st.st_mode))  printf("directory\n");
        else if (S_ISLNK(st.st_mode))  printf("symbolic link\n");
        /* ... other types ... */
    
        /* print the mode in symbolic form */
        /* print the timestamps */
        /* ... */
        return 0;
    }
    Test the program on a regular file, a directory, a symbolic link, a device file, and a FIFO.
  8. Reflect. In your own words, explain the difference between a file's logical size and its allocated size, and give one situation in which the difference is important. Then describe one situation in your own experience where a file's metadata was important — perhaps the modification time was used by a build system, or the permission bits determined whether a program could run, or the access time was used by a backup tool. What was the situation, and what would have gone wrong if the metadata had been incorrect?

10 Summary

  • File metadata is the information a file system maintains about a file, excluding its contents. It includes the file's identity, type, ownership, permissions, size, timestamps, and other attributes.
  • The metadata is exposed to user space through the stat family of system calls, which fill in a struct stat structure. The three calls — stat, lstat, and fstat — differ in how they interpret the argument: stat follows symbolic links, lstat does not, and fstat operates on an open file descriptor.
  • A file's identity is the pair (st_dev, st_ino): the device and the inode number within that device. The pair is used to uniquely identify a file, because inode numbers are meaningful only within a single file system.
  • The link count (st_nlink) is the number of hard links to the file. The inode is freed only when the link count reaches zero and no process has the file open.
  • The mode bits (st_mode) encode the file's type and its permission bits. The types are regular file, directory, symbolic link, character device, block device, FIFO, and socket. The permission bits are the owner, group, and other triplets, plus the set-user-ID, set-group-ID, and sticky bits.
  • The permission-check algorithm checks the effective UID and GID against the file's owner and group. Root bypasses the checks (with two exceptions). Otherwise, the first matching category — owner, group, or other — determines the permissions used; the kernel applies exactly one set of bits, not all three.
  • The three timestamps record different events: atime is the time of the last read, mtime is the time of the last content modification, and ctime is the time of the last metadata change (including content changes). The birthtime (creation time) is available on modern file systems through statx.
  • Linux supports three policies for updating atime: strictatime (every read), relatime (the default; only when the current atime is older than mtime or ctime, or after a day), and noatime (never). The choice balances performance and the usefulness of the field.
  • A file's logical size (st_size) is the number of bytes in the file; its allocated size (st_blocks × 512) is the space it occupies on disk. The two can differ because of block alignment or because the file is sparse — a file with holes, whose logical size includes regions that contain no data and are not allocated on disk.
  • Extended attributes (xattrs) are named key-value pairs associated with a file, in namespaces such as user., trusted., security., and system.. They are used for ACLs, security labels, file capabilities, file-system bookkeeping, and application metadata.
  • Access control lists (ACLs) extend the classical permission model with entries for specific users and groups. They are stored in the system.posix_acl_access extended attribute, and they interact with the classical permission bits in ways that are worth understanding.
  • The metadata lives in the file's inode (or its equivalent in modern file systems). The inode's structure and lifetime are the subject of Lesson 10.3; the strategies for laying out the metadata and for keeping it consistent in the face of crashes are the subject of Lessons 10.5 and 10.6.

Looking ahead

This lesson presented file metadata: the stat interface, the fields of the structure, the permission-check algorithm, the timestamps and their semantics, the size and block counts, and the extended attributes and ACLs that modern file systems support. It referred several times to the inode — the file system's own structure that holds the metadata — but did not present it in detail. Lesson 10.3 presents the inode itself: its structure, its fields, its role in path resolution, its lifetime, and the operations that manipulate it. The lesson examines the classical Unix inode, the changes introduced by the Berkeley Fast File System, and the modern inode structures used in ext4, XFS, Btrfs, and ZFS. It closes Unit 10's treatment of the file's identity and metadata, and it prepares the way for the discussion of directory organization (Lesson 10.4) and file allocation (Lesson 10.5).

11 References

  • Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 13, Section 13.1, and Chapter 14, Section 14.2.
  • Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapters 39 and 40. Available online at ostep.org.
  • Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 4, Sections 4.1 and 4.3.
  • Kerrisk, M. The Linux Programming Interface. No Starch Press, 2010. Chapters 15 and 16. The definitive reference for the stat structure, its fields, and Linux's extended attributes.
  • Bryant, R. E., & O'Hallaron, D. R. Computer Systems: A Programmer's Perspective, 3rd ed., Chapter 10, System-Level I/O.
  • McKusick, M. K., Joy, W. N., Leffler, S. J., & Fabry, R. S. A Fast File System for UNIX. ACM Transactions on Computer Systems, 1984. The paper that refined the inode structure and the metadata layout.
  • Ritchie, D. M., & Thompson, K. The UNIX Time-Sharing System. Communications of the ACM, 1974. The original paper that established the Unix file and metadata model.
  • IEEE Std 1003.1-2017 (POSIX.1-2017). The specification of the struct stat fields and the stat family of system calls.
  • Linux man-pages: man 2 stat, man 2 lstat, man 2 fstat, man 2 statx, man 2 chmod, man 2 chown, man 2 utime, man 2 utimensat, man 2 truncate, man 7 xattr, man 5 acl, man 5 proc, man 1 stat, man 1 getfattr, man 1 setfattr, man 1 getfacl, man 1 setfacl.
👤