Module 4 · Unit 10 · Lesson 2 of 7
File Metadata
Everything about a file except its contents. Metadata is the information the file system keeps about a file — its identity, its type, its owner, its permissions, its size, and the times at which it was created, modified, and accessed. This lesson presents the standard metadata interface, the semantics of each field, and the way the file system stores and updates it.
01 Introduction
Lesson 10.1 presented the file and directory abstractions: a file as a named, persistent sequence of bytes, and a directory as a mapping from names to files. It introduced the idea that a file has attributes — information about the file that is separate from its contents — and it noted that those attributes are stored in the file system's metadata structures, not in the file itself. This lesson examines those attributes in detail.
File metadata is the collection of
information that a file system maintains about a file:
everything except the file's data. The distinction is
fundamental. When a program calls open, the
file system reads the metadata to check permissions and to
find the file's contents. When a program calls
stat, the file system reads the metadata and
returns it without touching the contents at all. When a
file is modified, the file system updates the metadata
(the modification time, the size, the block count) as well
as the contents. The metadata is a first-class part of the
file, and its correctness and its performance matter as
much as the contents'.
The metadata is significant for three reasons. First, it is what makes a file more than a sequence of bytes: the metadata gives a file its identity (its inode number), its name (through the directory), its type, its owner, and its permissions. Second, the metadata is what the file system uses to find the file's contents: the pointers to the file's data blocks are stored in the metadata, not in the data itself. Third, the metadata is what the kernel consults on every access: the permission check, the dirty-flag check, the timestamp update, and the lookup of the file's data blocks all involve reading or writing the metadata.
This lesson presents the metadata in the order that a
program encounters it: the standard stat
interface (which exposes the metadata to user space), the
file's identity (device and inode number, link count), the
file's type and permission bits, the ownership fields, the
three timestamps and their semantics, the file's size and
block count (including the special case of sparse files),
and the extended attributes and access control lists that
modern file systems provide. It closes with a discussion
of where the metadata lives — the inode, which is the
subject of Lesson 10.3 — and of the cost of metadata
updates, which is the motivation for journaling (Lesson
10.6) and for the design of modern file systems (Lesson
10.7).
Prerequisites
This lesson builds directly on Lesson 10.1. The file and
directory abstractions, the distinction between a file's
name and its identity, and the standard file operations
are all assumed. The concept of an inode was introduced
in Lesson 10.1 as the file's metadata structure; this
lesson examines the metadata's contents, and
Lesson 10.3 examines the inode's
structure and lifetime. Familiarity with the
Unix permission bits (introduced in section 4.11 of
Lesson 10.1) is assumed, and the discussion of the
permission-check algorithm in section 4.5 extends that
material. No prior knowledge of the stat
interface, timestamps, sparse files, extended
attributes, or ACLs is required; all are introduced
here. The interactive activity uses the standard Unix
tools (stat, ls,
touch, getfattr,
getfacl), so familiarity with the Linux
command line from Lesson 2.2 is helpful.
02 Learning objectives
By the end of this lesson you will be able to
- Describe the standard metadata interface and identify the fields of the
struct statstructure. - Explain file identity (device and inode number) and the role of the link count.
- Describe the file type and mode bits and trace the kernel's permission-check algorithm.
- Distinguish the three timestamps (
atime,mtime,ctime) and explain when each is updated. - Explain sparse files, extended attributes, and access control lists, and describe where metadata lives in the file system.
This lesson contributes to U10.2 and U10.3, and to course learning outcomes CLO1, CLO5, and CLO6.
03 Required reading
- Operating System Concepts — Chapter 13, Section 13.1, File Concept, and Chapter 14, Section 14.2, File-System Implementation. The standard treatment of file attributes and of the inode as the container of metadata.
-
OSTEP — Chapter 39, Files and
Directories, and Chapter 40, File System
Implementation. OSTEP's treatment of the
statinterface and of the inode as the metadata structure is the clearest available account, and it is the basis for the worked example in this lesson. - Tanenbaum & Bos — Modern Operating Systems, Chapter 4, Sections 4.1 and 4.3, Files and File-System Implementation. The treatment of file attributes and the role of the inode.
- Silberschatz et al. — Chapter 13, Section 13.1, and Chapter 14, Section 14.2. Read the discussion of the attributes and of the inode's fields.
-
Kerrisk, M. — The Linux Programming
Interface, Chapter 15, File Attributes,
and Chapter 16, Extended Attributes. The
definitive reference for the
statstructure, its fields, and the Linux-specific extensions. - Bryant, R. E., & O'Hallaron, D. R. — Computer Systems: A Programmer's Perspective, 3rd ed., Chapter 10, System-Level I/O, and Chapter 9, Virtual Memory, section 9.10. The programmer's view of file metadata and of the interaction between the file system and the page cache.
- McKusick, M. K., Joy, W. N., Leffler, S. J., & Fabry, R. S. — A Fast File System for UNIX. ACM Transactions on Computer Systems, 1984. The paper that refined the inode structure and the metadata layout for performance, and that established many of the design principles still in use.
-
Linux man pages:
man 2 stat,man 2 lstat,man 2 fstat,man 2 statx,man 2 chmod,man 2 chown,man 2 utime,man 2 utimensat,man 2 truncate,man 7 symlink,man 7 xattr,man 5 acl,man 1 stat,man 1 getfattr,man 1 setfattr,man 1 getfacl,man 1 setfacl,man 5 proc(for the/proc/<pid>/statinterface).
Reading strategy
Read the OSTEP chapters first. Their treatment of the
inode as the container of file metadata, and of the
stat interface as the way user space sees
it, is the clearest available, and it establishes the
vocabulary for the rest of the lesson. Then read the
Kerrisk chapters for the definitive reference on the
struct stat fields and on Linux's extended
attributes. The man 2 stat page is worth
reading carefully: it documents each field, its units,
and the caveats that surround some of them (particularly
the timestamps, which have subtle semantics). Pay
particular attention to the timestamps: they are the
part of the metadata whose semantics are most often
misunderstood, and the kernel's policies for updating
them (the relatime, noatime,
and strictatime mount options) have a
significant effect on performance.
04 Core content
4.1 What is file metadata?
File metadata is the collection of
information that a file system maintains about a file,
excluding the file's contents. It includes the file's
identity, its type, its ownership, its permissions, its
size, its timestamps, and any other attributes the file
system supports. On Unix-like systems, the metadata is
stored in the file's inode (Lesson
10.3), and it is exposed to user space through the
stat family of system calls.
The metadata is not a separate entity from the file: it is the file, in the sense that the file's identity is its inode number and its contents are located through the pointers in the metadata. When you delete a file, you are removing the directory entry that refers to the metadata's inode; the metadata itself is freed when the last reference to it is gone. When you rename a file, you are changing the directory entry that refers to the inode; the metadata is unaffected. When you modify a file, you are changing the contents, and the metadata is updated to reflect the change (the modification time, the size, and the block count are all adjusted).
The metadata's layout and its relationship with the file's data are central to the file system's design. On classical Unix file systems, the metadata is stored in a fixed-size structure (the inode) whose location is determined by the file's inode number, and the file's data blocks are located through pointers stored in the inode. On modern file systems, the metadata may be stored in a more flexible structure — a B-tree, a log, or a copy-on-write tree — but the principle is the same: the metadata records the file's attributes and provides the pointers to the file's contents.
4.2 The stat interface
The metadata is exposed to user space through the
stat family of system calls. The calls fill in
a struct stat structure, which is the
user-space view of the file's metadata. On Linux, the
structure has the following fields:
struct stat {
dev_t st_dev; /* device containing the file */
ino_t st_ino; /* inode number (file identity) */
mode_t st_mode; /* file type and permission bits */
nlink_t st_nlink; /* number of hard links */
uid_t st_uid; /* user ID of the owner */
gid_t st_gid; /* group ID of the group */
dev_t st_rdev; /* device ID (for device files) */
off_t st_size; /* total size in bytes */
blksize_t st_blksize; /* preferred block size for I/O */
blkcnt_t st_blocks; /* number of 512-byte blocks allocated */
struct timespec st_atim; /* time of last access */
struct timespec st_mtim; /* time of last modification */
struct timespec st_ctim; /* time of last status change */
/* Linux also provides st_birthtim (creation time) in statx */
};
The three calls differ in how they interpret the path:
-
stat(path, buf)— fills in the structure with the metadata of the file thatpathrefers to, following symbolic links. Ifpathis a symbolic link, the metadata returned is that of the target, not the link. -
lstat(path, buf)— the same, but does not follow symbolic links. Ifpathis a symbolic link, the metadata returned is that of the link itself (its size is the length of the pathname it contains, and its type is symbolic link). -
fstat(fd, buf)— takes an open file descriptor rather than a path, and fills in the structure with the metadata of the file the descriptor refers to. Useful for files that have been opened but whose path is not known, or that may have been renamed since they were opened.
Linux also provides statx, an extended
version of stat that exposes additional
fields, including the file's creation time
(stx_btime), the file's attributes
(stx_attributes), and the file's
mount ID. The extended interface is the subject of
section 4.8.
struct stat interface and
the metadata it exposes. The structure is the user-space
view of a file's inode. Its fields group into five
purposes: identity, type and mode, ownership, size, and
timestamps. The kernel fills the structure from the
file's inode on disk, and the structure is a snapshot of
the inode's contents at the moment of the call.
4.3 File identity
A file's identity is the pair
(st_dev, st_ino): the device that contains
the file and the inode number within that device. The
pair is necessary because an inode number is meaningful
only within a single file system; the same inode number
can refer to different files on different devices.
Together, the pair uniquely identifies the file within
the entire system.
Programs that need to determine whether two paths refer
to the same file compare their (st_dev, st_ino)
pairs. This is what find does to detect hard
links to the same file, what rsync does to
detect files that have been renamed, and what the
kernel's directory-entry cache uses to recognise when
two paths resolve to the same file.
The st_nlink field records the
link count: the number of hard links to
the file. When a hard link is created, the count is
incremented; when a hard link is removed, the count is
decremented. The file's inode is freed only when the count
reaches zero and no process has the file open. The link
count is a crucial part of the file system's bookkeeping:
it is what allows the file to remain accessible through
other names after one of its names is deleted, and what
ensures that the file's resources are reclaimed only when
the last reference is gone.
On Unix, the link count for a regular file starts at 1
when the file is created (there is one name: the file
itself). For a directory, the link count starts at 2 —
one for the directory's own name in its parent, and one
for the . entry inside it. Each subdirectory
adds one more to the count, because each subdirectory
contains a .. entry that refers to its
parent. This is why a directory's link count is one plus
the number of its immediate subdirectories.
4.4 File type and mode bits
The st_mode field encodes two pieces of
information: the file's type and its
permission bits. The type is stored in
the high bits of the field, and the permission bits in
the low nine bits. The standard types are:
| Type | Symbol | Description |
|---|---|---|
| Regular file | S_IFREG |
An ordinary file: a sequence of bytes with no special meaning to the file system. |
| Directory | S_IFDIR |
A file whose contents are a sequence of directory entries. |
| Symbolic link | S_IFLNK |
A file whose contents are a pathname; dereferenced on access unless lstat is used. |
| Character device | S_IFCHR |
A device accessed as a stream of bytes: a terminal, a serial port, /dev/null. |
| Block device | S_IFBLK |
A device accessed in fixed-size blocks: a disk, a partition. |
| FIFO (named pipe) | S_IFIFO |
A named channel for inter-process communication. |
| Socket | S_IFSOCK |
A Unix-domain socket bound to a pathname. |
The permission bits occupy the low nine bits of
st_mode and are organised into three
triplets: the owner's permissions, the group's
permissions, and the permissions for everyone else. Each
triplet contains a read bit, a write bit, and an execute
bit. The symbolic constants are S_IRUSR,
S_IWUSR, S_IXUSR for the
owner, S_IRGRP, S_IWGRP,
S_IXGRP for the group, and
S_IROTH, S_IWOTH,
S_IXOTH for others.
The meaning of the execute bit depends on the file's type. For a regular file, the execute bit grants permission to execute the file as a program. For a directory, the execute bit grants permission to search the directory — that is, to look up names within it. A directory with read permission but not execute permission can be listed (the names can be read), but the entries cannot be accessed (the names cannot be resolved to inodes). A directory with execute permission but not read permission can be traversed (a path through it can be resolved) but its contents cannot be listed. The combination is unusual but legal.
Three additional bits are stored in the same field, in positions adjacent to the permission bits:
-
Set-user-ID (SUID) — When an executable
file with the SUID bit is run, the process runs with the
privileges of the file's owner, not the user who
invoked it. This is how programs like
passwdare able to modify/etc/shadow, which is writable only by root. - Set-group-ID (SGID) — The same, but for the group. On a directory, the SGID bit has a different meaning: files created in the directory inherit the directory's group, not the creating user's group. This is used for shared project directories.
-
Sticky bit — On a directory, the sticky
bit restricts deletion: a file in a sticky directory can
be deleted only by its owner, the directory's owner, or
root. This is used for directories like
/tmp, where many users create files but should not be able to delete each other's.
4.5 Ownership and the permission-check algorithm
The st_uid and st_gid fields
record the file's owner and group. They are used for the
permission check — the kernel compares the calling
process's effective user and group IDs against the
file's — and for accounting (which user owns how much
data).
The permission check is performed on every access, and its algorithm is specific enough to be worth stating precisely. When a process tries to access a file, the kernel applies the following rules, in order:
- If the process's effective UID is 0 (root), the access is allowed. Root bypasses the permission checks entirely, with two important exceptions: executing a regular file requires at least one execute bit to be set, and writing to a file in a read-only file system is not allowed. The exceptions exist so that root cannot accidentally execute a non-executable file or modify a read-only file system.
-
Otherwise, if the process's effective UID equals
the file's
st_uid, the owner's permission bits (the high triplet of the nine bits) are used. The group and other bits are ignored, even if they would grant more permission. -
Otherwise, if the process's effective GID (or
one of its supplementary GIDs) equals the file's
st_gid, the group's permission bits (the middle triplet) are used. - Otherwise, the other permission bits (the low triplet) are used.
The order is significant. The owner check comes before
the group check, and the group check comes before the
other check. The kernel applies one set of bits,
not all three: if the process is the owner, the owner's
bits apply even if the other bits would grant more
permission. This is why a file with mode
---rwxrwx (no permission for the owner,
full permission for everyone else) is inaccessible to
the owner, but accessible to every other user.
The check is performed against the process's effective UID and GID, not its real ones. The distinction matters for the set-user-ID and set-group-ID mechanisms: when a process runs a set-user-ID binary, its effective UID is changed to the file owner's UID, and the permission check uses the changed value. This is what allows the program to access files the invoking user could not otherwise access.
4.6 Timestamps
The metadata contains three or four timestamps, each updated under different conditions. Their semantics are subtle, and they are the part of the metadata most often misunderstood.
st_atim— access time (atime)-
The time of the file's last read. Updated when
a program reads the file's contents, or when a program
lists a directory's contents. Not updated when the
file's metadata is read (as with
stat), except under thestrictatimepolicy. The access time is useful for archival tools (which can identify files that have not been read for a long time) and for intrusion detection (which can identify files that have been read unexpectedly), but it is expensive to maintain, because it must be updated on every read. st_mtim— modification time (mtime)-
The time of the file's last content
modification. Updated when a program writes to the
file's contents, truncates the file, or creates a
new file. Not updated when the file's metadata
is changed (changing the permission bits or the owner
does not update the mtime). The modification time is
the timestamp that most tools use as the file's
"version" — it is what
makecompares to decide whether a target needs to be rebuilt, and whatrsynccompares to decide whether a file has changed. st_ctim— status change time (ctime)- The time of the file's last metadata change. Updated when the inode is modified: when the permission bits are changed, when the owner is changed, when the link count is changed, when the file's size changes, when the file is renamed, and when the file's content is modified (because content modification also updates the inode's size and modification time). The ctime is therefore a superset of the mtime: every mtime update also updates the ctime, but the ctime can be updated without the mtime (a permission change updates the ctime alone).
st_birthtim— creation time (birthtime)-
The time at which the file was created. Not all file
systems support it, and not all kernels expose it:
classical Unix file systems did not track the creation
time, and the classical
struct stathas no field for it. Modern file systems (ext4, XFS, Btrfs, ZFS, APFS, NTFS) do track the creation time, and Linux exposes it through the extendedstatxinterface asstx_btime. The field is sometimes called the birthtime to distinguish it from the ctime.
The three classical timestamps — atime, mtime, and ctime — are the ones that every Unix system provides. The birthtime is a modern addition. The distinctions between them are easy to confuse, and it is worth restating: mtime is the content, ctime is the metadata, atime is the read.
atime tracks reads,
mtime tracks content changes, and
ctime tracks any metadata change — including
content changes, which are metadata changes as well. The
birthtime is set once at creation and
never changes.
The timestamps are also the subject of a Linux-specific
optimisation that is worth understanding. Updating
atime on every read is expensive, because it
turns every read into a write to the inode. The write is
small — a few bytes — but it requires the inode to be
located, the disk block to be read, modified, and written
back, and it can be a significant fraction of the cost of
a read that would otherwise be served entirely from the
page cache. To avoid this cost, Linux provides three
mount options that control when atime is
updated:
-
strictatime— Updateatimeon every read. This is the classical behaviour, and it is what the POSIX standard requires. It is rarely used because of its cost. -
relatime— Updateatimeonly if the file's currentatimeis older than itsmtimeorctime, or if more than a day has passed since the last update. This is the default on most Linux systems. It preserves the timestamps' usefulness for most purposes while eliminating the cost of updating them on every read. -
noatime— Never updateatime. This is the fastest option, but it makesatimeuseless. It is used on systems where performance matters more than the access-time information, and on systems where theatimefield is not needed (many modern systems usenoatimeon their root file systems).
The choice of policy is a trade-off between performance
and the usefulness of the atime field. Most
modern systems choose relatime as a
compromise: it updates the field often enough for
archival tools and intrusion detection, but rarely
enough to avoid the cost on the common case of a file
that is read repeatedly.
4.7 Size and block counts
The metadata records two measures of the file's size:
st_size, the logical size in bytes, and
st_blocks, the number of 512-byte blocks
actually allocated on disk. The two are usually related
(the block count is at least the logical size divided by
512), but they can differ in both directions.
The most common reason for a difference is
block alignment. A file system allocates
space in fixed-size blocks (typically 4 KB on ext4, 8 KB
or larger on some others), and the last block of a file
may be only partially used. A file with a logical size of
1 byte occupies one full block on disk, so
st_size is 1 and st_blocks is
8 (one 4 KB block divided into 512-byte units). The
difference is the file system's internal fragmentation,
and it is the price of the file system's block
granularity.
The other reason is the opposite: a file's
st_size can be much larger than the space it
actually occupies. This is the case for a
sparse file, a file whose logical size
includes "holes" — regions that contain no data. A sparse
file is created when a program seeks past the end of the
file and writes data there, or when a program explicitly
creates a sparse file with ftruncate or
lseek. The file system does not allocate
blocks for the holes: the metadata's block pointers are
NULL for the hole's regions, and the file system returns
zeros when a program reads them. The result is a file
whose st_size may be many gigabytes but whose
st_blocks may be only a few megabytes.
Sparse files are useful for several purposes. A virtual machine's disk image is typically sparse: the image's logical size is the size of the virtual disk, but the space actually allocated is only the space the guest operating system has written. A database's file might be sparse: the file is pre-allocated to a maximum size, but the space for unused records is not allocated. A scientific computation's output file might be sparse: the file is large, but only the regions the computation actually produced contain data.
The handling of sparse files is one of the differences between file systems. The classical Unix file system and its descendants (ext2, ext3, ext4) handle sparse files by leaving the block pointers for a hole empty. Modern file systems (XFS, Btrfs, ZFS) handle them more efficiently, using extents — a range of contiguous blocks with a single descriptor — which allows a hole to be represented by a single descriptor rather than by the absence of many pointers. And some file systems support hole punching: an operation that creates a hole in an existing file, freeing the blocks that were allocated there. Hole punching is used in files that have been written and then have regions marked as "do not care", such as the tail of a virtual machine's disk image after a snapshot.
4.8 Extended attributes
The classical struct stat exposes the
metadata that the POSIX standard specifies, but modern
file systems support additional metadata that does not
fit in the classical structure. This additional metadata
is stored in extended attributes
(xattrs), which are named key-value pairs associated with
a file.
An extended attribute has a name and a value. The name is
a string in a namespaced format: on Linux, the namespaces
are user. (for user-defined attributes),
trusted. (for attributes visible only to
privileged processes), security. (for
security attributes such as SELinux labels and
capabilities), and system. (for system
attributes such as POSIX ACLs). The value is a byte
string whose meaning is determined by the namespace and
by the application that uses it.
Extended attributes are used for several purposes.
-
Access control lists. On Linux, POSIX
ACLs are stored as extended attributes in the
system.namespace. The classical permission bits remain in the inode; the ACL extends them with additional entries. -
Security labels. SELinux and AppArmor
store their security context in
security.attributes. -
File capabilities. The
security.capabilityattribute stores the capabilities that a program should have when it is executed, as an alternative to the set-user-ID mechanism. - File system features. A file system may use extended attributes for its own bookkeeping: for example, to record the file's checksums, its encryption keys, or its compression flags.
-
Application metadata. An application
may store arbitrary metadata in the
user.namespace: the source URL of a downloaded file, the encoding of a document, the thumbnail of an image.
The support for extended attributes varies between file
systems. On Linux, ext4 supports extended attributes up
to a limited total size (typically a few kilobytes per
inode), and they are stored either inline in the inode
or in a separate block pointed to by the inode. XFS, Btrfs,
and ZFS support extended attributes with larger limits.
The interface for manipulating them is
setxattr, getxattr,
listxattr, and removexattr, and
the command-line tools are setfattr and
getfattr.
4.9 Access control lists
The classical Unix permission model is a good fit for the majority of cases: a file has an owner, a group, and a set of permissions, and that is sufficient. But it has a fundamental limitation: it can express permissions for at most one group. A file that needs to be readable by three different groups, or by a specific list of users, cannot be expressed in the classical model without creating additional groups or duplicating the file.
Access control lists (ACLs) extend the model with a list of entries, each specifying a user or group and a set of permissions. The list is consulted after the classical permission bits: if the classical bits already allow the access, the ACL is not consulted; if they do not, the ACL is checked for an entry that grants the access.
A POSIX ACL has up to three kinds of entry:
-
User entries — one for the file's
owner (the
user::entry, which corresponds to the classical owner bits) and zero or more for specific users (theuser:alice:entries). -
Group entries — one for the file's
owning group (the
group::entry, which corresponds to the classical group bits) and zero or more for specific groups (thegroup:staff:entries). -
Other entries — one for everyone else
(the
other::entry, which corresponds to the classical other bits). - Mask entry — a special entry that limits the permissions granted by the user and group entries. The mask is intersected with the permissions in each entry, so the effective permission of a user entry is the entry's permission AND the mask.
The ACL's entries are stored in an extended attribute
(system.posix_acl_access), and the classical
permission bits continue to exist alongside them. The
classical bits and the ACL are kept in sync: changing
the classical bits with chmod updates the
ACL's owner, group, and other entries, and changing those
entries in the ACL updates the classical bits. This is
why chmod works on a file with an ACL, and
why the interaction between the two is more subtle than
it first appears.
ACLs are a powerful tool, but they are also a source of
complexity. A file with an ACL behaves differently from a
file without one: the permission check requires reading
the ACL, the ls -l output shows a
+ after the mode to indicate the presence of
an ACL, and the classical permission bits have a
different meaning (they are the maximum, not the actual
permissions). Most systems therefore use ACLs sparingly,
for the cases where the classical model is genuinely
insufficient.
4.10 Where metadata lives
The metadata is stored in the file system's own structures, separate from the file's data. On classical Unix file systems, this structure is the inode, a fixed-size record whose location is determined by the file's inode number. The inode contains all the file's metadata — its type, its permissions, its owner, its size, its timestamps, and the pointers to its data blocks — and it is read and written as a unit.
Modern file systems have relaxed the fixed-size,
fixed-location assumption of the classical inode.
Ext4 still uses fixed-size inodes but supports extended
attributes that spill into additional blocks. XFS uses
variable-size inodes that can grow as needed. Btrfs and
ZFS store metadata in B-trees and copy-on-write trees,
with no separate inode structure at all: the file's
metadata is a node in a tree, and the tree is updated
atomically. The struct stat interface is
preserved across these changes — the metadata is still
exposed to user space in the same way — but the internal
representation is different.
The separation of metadata from data has a cost. Reading
a file requires the file system to read the file's
metadata (to find the data blocks) and then read the data
blocks themselves. For a small file, the metadata read
dominates; for a large file, the data reads dominate. And
writing a file requires updating both the data blocks and
the metadata (the size, the modification time, the block
pointers), so a single write can involve
several disk accesses to different parts of the file
system. This is the subject of Lesson 10.5 (file
allocation) and Lesson 10.6 (journaling), which examine
the strategies for laying out the metadata and for
keeping it consistent in the face of crashes.
4.11 Common misconceptions
"Metadata is optional."
Metadata is not optional: it is what makes a file a file. Without the metadata, the file has no identity, no name, no owner, no permissions, and no way to locate its contents. Every file has metadata, and the metadata is as much a part of the file as its contents.
"st_size is the space the file occupies
on disk."
st_size is the logical size in bytes;
st_blocks × 512 is the space actually
allocated. The two can differ: a file with a logical
size of 1 byte occupies at least one block, and a
sparse file can have a logical size much larger than
its allocated space.
"mtime and ctime are the
same."
They are not. mtime is the time of the
last content modification; ctime is the
time of the last metadata change. Every
mtime update also updates
ctime, but not vice versa: a
chmod or chown updates
ctime alone.
"The owner of a file always has all permissions."
The owner is subject to the owner's permission bits,
which can be set to deny any permission. A file with
mode ---rwxrwx is inaccessible to its
owner but accessible to everyone else.
"Root can do anything."
Root bypasses the permission checks, but not unconditionally: root cannot execute a file with no execute bits set, and root cannot write to a read-only file system. The exceptions exist for safety.
"A hard link and a copy are the same."
A copy is a new file with its own metadata and its own data blocks; modifying the copy does not affect the original. A hard link is the same file, reached through a different name; modifying the file through either name affects the other. The two have different link counts and different inode numbers.
"Extended attributes are part of the file's contents."
Extended attributes are metadata, not contents. They
are stored in the inode (or in blocks pointed to by
it), they are read with a different interface, and
they are not part of the byte stream that
read and write see.
"The struct stat is the file's
metadata."
The struct stat is the user-space
view of the metadata; the metadata itself
lives in the inode (or its equivalent). The
structure is a copy, filled in by the kernel at the
moment of the call, and changes to the file's
metadata afterward are not reflected in the copy.
05 Worked example
The following example interprets the metadata of several files, traces the permission-check algorithm through a specific case, and demonstrates the behaviour of sparse files.
Interpreting the metadata of a file
Suppose a program calls stat on a file and
receives the following structure (fields abbreviated):
st_dev = 0x0801 /* device 8, minor 1 */
st_ino = 262147
st_mode = 0x81A4 /* 0100644 in octal */
st_nlink = 3
st_uid = 1000 /* alice */
st_gid = 100 /* staff */
st_rdev = 0
st_size = 40960 /* 40 KB */
st_blksize = 4096
st_blocks = 80 /* 80 × 512 = 40 KB */
st_atim = 1712345678.123456789
st_mtim = 1712345000.987654321
st_ctim = 1712345000.987654321
The fields can be interpreted as follows.
-
Identity.
(st_dev, st_ino) = (0x0801, 262147)uniquely identifies the file within the system.st_nlink = 3means there are three hard links to the file, so the file has three names. -
Type and mode. The mode
0x81A4is0100644in octal. The010prefix isS_IFREG, so the file is a regular file. The low three octal digits are644, so the owner has read and write, and the group and others have read only. - Ownership. The file is owned by user ID 1000 (alice) and group ID 100 (staff).
-
Size. The logical size is 40 KB, and
the allocated size is 80 blocks × 512 bytes = 40 KB.
The two match, so the file has no holes and occupies
exactly as much space on disk as its logical size
suggests. (Actually, 40 KB is a multiple of 4 KB, so
no partial block is involved; a file with a size of
40 KB + 1 byte would have the same
st_blocksbut a differentst_size.) -
Timestamps. The access time is about
11 minutes later than the modification time, which
means the file was read after it was written. The
modification and change times are identical, which
means the last operation on the file was a content
modification (not a metadata change alone). If a
chmodhad been run after the write, thectimwould be later than themtim.
Tracing the permission check
A process running as user ID 2000 (bob) tries to open
the file with O_WRONLY. Bob is not the
owner (UID 1000), but he is a member of group 100
(staff). The kernel's permission check proceeds as
follows:
- Is the process's effective UID 0? No (bob's UID is 2000, not 0). Root's bypass does not apply.
-
Is the process's effective UID equal to the
file's
st_uid? No (2000 ≠ 1000). The owner's permission bits are not used. -
Is the process's effective GID (or any
supplementary GID) equal to the file's
st_gid? Yes (bob is a member of group 100). The group's permission bits are used. -
Do the group's permission bits include the
requested access? The mode is
644, so the group's bits arer--: read permission only, no write permission. The requested access is write, so the check fails.
The open call returns -1
with errno set to EACCES.
Bob cannot write to the file, even though he can read
it.
Now suppose the file's mode is changed to
664 (owner and group read-write, others
read). The owner's bits are still rw-,
the group's bits become rw-, and the
others' bits stay r--. Bob's check now
reaches step 4 with the group's bits
rw-, which include the requested write
access, so the check succeeds. The open
call returns a file descriptor.
Finally, suppose the file is changed to mode
600 (owner read-write only). Bob's check
reaches step 3 with the group's bits
---, which do not include the requested
write access, so the check fails. Bob cannot write to
the file, even though he is a member of the file's
group. The group membership is not sufficient; the
group's bits must also grant the access.
Demonstrating sparse files
A program creates a file, seeks past the end, and writes a byte:
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
int main(void) {
int fd = open("sparse.bin", O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (fd < 0) { perror("open"); return 1; }
/* write one byte at offset 0 */
if (write(fd, "A", 1) != 1) { perror("write"); return 1; }
/* seek to 1 GB past the start */
if (lseek(fd, 1024L * 1024 * 1024, SEEK_SET) == -1) {
perror("lseek"); return 1;
}
/* write one byte at the new position */
if (write(fd, "B", 1) != 1) { perror("write"); return 1; }
close(fd);
return 0;
}
After the program runs, the file's metadata is examined:
$ stat sparse.bin
File: sparse.bin
Size: 1073741825 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 524291 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ alice) Gid: ( 100/ staff)
Access: 2025-04-05 10:15:32.123456789 +0000
Modify: 2025-04-05 10:15:32.123456789 +0000
Change: 2025-04-05 10:15:32.123456789 +0000
Birth: 2025-04-05 10:15:32.123456789 +0000
The file's logical size is 1073741825 bytes
— one byte more than 1 GB. Its allocated size is
8 blocks of 512 bytes, or 4 KB. The
difference is the hole: the file system has allocated
only the two blocks that contain the two bytes written
(one block for the first byte, one block for the last
byte), and the 1 GB between them is a hole. The hole
contains no data, and reading from it returns zeros
without any disk access.
The block count of 8 (one 4 KB block) is the minimum allocation for a file with data at two distant offsets: the file system allocates a block for the first byte and a block for the last byte, and the two blocks happen to fall in the same 4 KB unit — no, that cannot be, since the offsets are 1 GB apart. The block count of 8 means the file system has allocated 4 KB of disk for the file's two blocks. This illustrates the difference between the logical size and the physical size: the file appears to be 1 GB, but it occupies 4 KB on disk.
If the file is copied with a program that reads and
writes every byte — as cp does without the
--sparse option — the copy loses the
sparseness, because the program reads a stream of
bytes (mostly zeros) and writes them all, allocating
blocks for the entire 1 GB. If the file is copied with
a program that is aware of holes — as cp
--sparse or rsync --sparse is —
the copy preserves the sparseness by seeking past the
holes instead of writing zeros for them.
Comparing stat, lstat, and
fstat
Suppose a symbolic link link.txt points to
a regular file target.txt with a size of
1000 bytes. The three calls produce different results:
| Call | Argument | st_size |
st_mode type |
|---|---|---|---|
stat("link.txt", &buf) |
The link's path | 1000 | Regular file (the target) |
lstat("link.txt", &buf) |
The link's path | 8 | Symbolic link (the link itself) |
fstat(fd, &buf) |
An open FD for the target | 1000 | Regular file (the target) |
The stat call follows the link, so it
returns the target's metadata: size 1000, type regular
file. The lstat call does not follow the
link, so it returns the link's metadata: size 8 (the
length of the pathname the link contains, assuming the
target's path is 8 characters long), type symbolic
link. The fstat call operates on the
open file descriptor, which was obtained by opening
the target (the open call follows the link),
so it returns the target's metadata.
The distinction matters for programs that need to
distinguish between a symbolic link and its target: a
backup program that wants to preserve symbolic links
rather than copy their targets, a security tool that
wants to check whether a path is a symbolic link
before following it, and a shell that implements the
-L and -h test operators for
file existence.
06 Interactive activity
Activity · explore file metadata on a live system
The goal of this activity is to use the Linux command line to explore file metadata, to observe the timestamps' behaviour, and to demonstrate the permission-check algorithm and the effects of sparse files. Work through the steps in your course VM.
-
Inspect file metadata.
Use the
statcommand to see a file's metadata, and identify each field.
For each file, identify: its inode number, its device, its type, its mode, its owner and group, its size, its block count, and its three timestamps. Note the difference between the logical size (stat /etc/passwd stat /bin/ls stat /tmpSize) and the allocated size (Blocks × 512). -
Observe the timestamps.
Create a file and observe how its timestamps change
as you perform different operations.
After each step, observe which timestamps changed. The pattern should be: a read updates only# Create a file echo "hello" > /tmp/timestamps.txt stat /tmp/timestamps.txt # Read the file cat /tmp/timestamps.txt sleep 2 stat /tmp/timestamps.txt # atime should have changed # Write to the file echo "world" >> /tmp/timestamps.txt sleep 2 stat /tmp/timestamps.txt # mtime and ctime should have changed # Change the permissions chmod 600 /tmp/timestamps.txt sleep 2 stat /tmp/timestamps.txt # only ctime should have changedAccess; a write updates bothModifyandChange; achmodupdates onlyChange. -
Examine the mount options for atime.
Check the mount options for the file system that
holds your test file.
Look for the optionsfindmnt -T /tmp -o TARGET,SOURCE,FSTYPE,OPTIONS cat /proc/self/mounts | grep -E 'ext4|xfs|btrfs'relatime,noatime, orstrictatime. Which is in effect? If the file system is mounted withrelatime(the common default), the atime may not update on every read: it updates only when the current atime is older than the mtime or ctime, or when more than a day has passed. Observe whether your test in the previous step actually updated the atime. -
Test the permission-check algorithm.
Create a file and set its mode to various values, and
observe the effect on a second user (if you have
sudoand access to a second user such asnobody).
At each step, identify which set of permission bits theecho "secret" > /tmp/perms.txt chmod 640 /tmp/perms.txt ls -l /tmp/perms.txt # Attempt to read as another user sudo -u nobody cat /tmp/perms.txt 2>&1 # Change to 644 — the group's read permission becomes world's chmod 644 /tmp/perms.txt sudo -u nobody cat /tmp/perms.txt # Change to 600 — only the owner can read chmod 600 /tmp/perms.txt sudo -u nobody cat /tmp/perms.txt 2>&1 # Change to 604 — everyone except the group can read chmod 604 /tmp/perms.txt sudo -u nobody cat /tmp/perms.txtnobodyuser matches, and whether the check succeeds. Thenobodyuser is neither the owner nor (usually) a member of the file's group, so the "other" bits are used. -
Demonstrate sparse files.
Write a small program that creates a sparse file, and
compare its logical and physical sizes.
#include <fcntl.h> #include <stdio.h> #include <unistd.h> int main(void) { int fd = open("sparse.bin", O_WRONLY | O_CREAT | O_TRUNC, 0644); if (fd < 0) { perror("open"); return 1; } write(fd, "A", 1); lseek(fd, 1024L * 1024 * 1024, SEEK_SET); write(fd, "B", 1); close(fd); return 0; }
Compare the three outputs. The logical size is 1 GB + 1 byte; the disk usage should be a few kilobytes. Then copy the file withgcc -o sparse sparse.c ./sparse stat sparse.bin du -h sparse.bin # disk usage (allocated) ls -lh sparse.bin # logical sizecpand withcp --sparse=always, and compare the copies' disk usage:
The dense copy loses the sparseness; the sparse copy preserves it.cp sparse.bin dense.bin cp --sparse=always sparse.bin sparse-copy.bin du -h sparse.bin dense.bin sparse-copy.bin -
Inspect extended attributes.
On Linux, the
getfattrandsetfattrcommands manipulate extended attributes.
Note that the# Create a file echo "hello" > /tmp/xattr-demo.txt # List its extended attributes getfattr -d /tmp/xattr-demo.txt # Set a user attribute setfattr -n user.comment -v "this is a demo" /tmp/xattr-demo.txt getfattr -d /tmp/xattr-demo.txt # Read the attribute getfattr -n user.comment /tmp/xattr-demo.txt # Remove the attribute setfattr -x user.comment /tmp/xattr-demo.txtuser.namespace is readable and writable by the file's owner; the other namespaces require special privileges. Also check whether the file system supports extended attributes at all: ifsetfattrfails withEOPNOTSUPP, the file system does not. -
Inspect access control lists.
On Linux, the
getfaclandsetfaclcommands manipulate ACLs.
After adding the ACL entry, the# Create a file echo "shared" > /tmp/acl-demo.txt # Show the ACL getfacl /tmp/acl-demo.txt # Add an entry for a specific user sudo setfacl -m u:nobody:r /tmp/acl-demo.txt getfacl /tmp/acl-demo.txt ls -l /tmp/acl-demo.txt # note the + after the mode # Attempt to read as nobody sudo -u nobody cat /tmp/acl-demo.txt # Remove the ACL entry sudo setfacl -x u:nobody /tmp/acl-demo.txt getfacl /tmp/acl-demo.txtls -loutput shows a+after the mode, which indicates that the file has an ACL. Thegetfacloutput shows the ACL's entries, including themaskentry. Observe how the classical permission bits interact with the ACL entries. -
Observe the metadata of a sparse file system
operation.
The metadata is updated when a file's size changes.
Create a file, write to it, and observe the block
count grow.
After the append, the logical size increases by 10 MB, and the block count increases accordingly. After the truncate, the logical size drops to 1 byte, and the block count drops as well — the blocks that held the file's contents are freed.echo "a" > /tmp/grow.txt stat -c 'size=%s blocks=%b' /tmp/grow.txt # Append data dd if=/dev/zero bs=1M count=10 >> /tmp/grow.txt stat -c 'size=%s blocks=%b' /tmp/grow.txt # Truncate truncate -s 1 /tmp/grow.txt stat -c 'size=%s blocks=%b' /tmp/grow.txt -
Read the man pages.
Read the man pages that document the metadata
interface.
For each, write a sentence describing its main contribution to your understanding of file metadata. Theman 2 stat man 2 statx man 5 proc | grep -A 20 '/proc/\[pid\]/stat' man 1 statman 2 statpage documents each field of the structure; theman 2 statxpage documents the extended interface; the/proc/<pid>/statinterface is a related structure (for processes, not files) that uses the same conventions.
Reflect: the metadata is a surprisingly rich source of information about a file. It records not only what the file is but also how it has been used: when it was created, when it was last modified, when it was last read, and whether it is being read more or less often than it is written. This information is used by a wide range of tools: backup programs use the mtime to decide what to back up, archival tools use the atime to decide what to migrate, intrusion-detection tools use the ctime to detect unauthorized modifications, and the kernel itself uses the metadata to enforce permissions and to find the file's data. Which of these uses do you think is the most important, and which do you think is the most frequently overlooked? What would you do differently if you were designing the metadata interface today?
07 Knowledge check
Check your understanding
Question 1
Define file metadata and identify the fields of the
struct stat structure. What is the
relationship between the structure and the file's
inode?
Show answer
struct stat structure
is the user-space view of the metadata, filled in by
the kernel when a program calls stat,
lstat, or fstat. Its
fields are:
st_dev— the device containing the filest_ino— the inode number (the file's identity within the device)st_mode— the file's type and permission bitsst_nlink— the number of hard links to the filest_uid— the user ID of the ownerst_gid— the group ID of the groupst_rdev— the device ID, for device filesst_size— the logical size of the file in bytesst_blksize— the preferred block size for I/Ost_blocks— the number of 512-byte blocks allocatedst_atim— the time of last accessst_mtim— the time of last content modificationst_ctim— the time of last metadata changest_birthtim— the creation time, available on modern file systems throughstatx
stat,
lstat, and fstat — differ
in how they interpret the argument: stat
follows symbolic links, lstat does not,
and fstat operates on an open file
descriptor rather than a path.
Question 2
What is a file's identity, and why is the pair
(st_dev, st_ino) used rather than just
st_ino? What is the link count, and how
is it maintained?
Show answer
(st_dev, st_ino): the device that
contains the file and the inode number within that
device. The pair is necessary because an inode
number is meaningful only within a single file
system: the same inode number can refer to different
files on different devices, and the same file
cannot be uniquely identified by its inode number
alone if the system has multiple file systems. The
pair uniquely identifies the file within the entire
system, and it is what programs compare when they
need to determine whether two paths refer to the
same file. The link count
(st_nlink) is the number of hard links
to the file: the number of directory entries that
refer to the file's inode. It is incremented when a
hard link is created (with the link
system call, or with the ln command
without the -s flag) and decremented
when a hard link is removed (with the
unlink system call, or with
rm). The file's inode is freed only
when the link count reaches zero and no process has
the file open. This is what allows the file to
remain accessible through other names after one of
its names is deleted, and what ensures that the
file's resources are reclaimed only when the last
reference is gone. On Unix, a regular file's link
count starts at 1 when the file is created; a
directory's link count starts at 2 and grows by one
with each subdirectory, because each subdirectory
contains a .. entry that refers to its
parent.
Question 3
Describe the kernel's permission-check algorithm for file access. What is the order of the checks, and why does the order matter? What are the exceptions for the root user?
Show answer
- If the process's effective UID is 0 (root), the access is allowed, with two exceptions: executing a regular file requires at least one execute bit to be set, and writing to a file in a read-only file system is not allowed.
-
Otherwise, if the process's effective
UID equals the file's
st_uid, the owner's permission bits (the high triplet) are used. The group and other bits are ignored, even if they would grant more permission. -
Otherwise, if the process's effective
GID (or one of its supplementary GIDs) equals
the file's
st_gid, the group's permission bits (the middle triplet) are used. - Otherwise, the other permission bits (the low triplet) are used.
---rwxrwx
is inaccessible to its owner (the owner's bits
deny access, and the group and other bits are not
consulted) but accessible to everyone else. The
exceptions for root exist for safety: root cannot
accidentally execute a file with no execute bits
set, and root cannot write to a read-only file
system. The check is performed against the
process's effective UID and GID, not its
real ones. The distinction matters for the
set-user-ID and set-group-ID mechanisms, which
change the effective IDs so that a program can run
with the privileges of the file's owner.
Question 4
Distinguish the three timestamps
(atime, mtime,
ctime). For each, state what it records
and when it is updated. Give an example of an
operation that updates ctime but not
mtime.
Show answer
-
atime(access time) records the time of the file's last read. It is updated when a program reads the file's contents, or lists a directory's contents. It is not updated when the file's metadata is read (as withstat), except under thestrictatimemount option. -
mtime(modification time) records the time of the file's last content modification. It is updated when a program writes to the file's contents, truncates the file, or creates a new file. It is not updated when only the file's metadata changes. -
ctime(status change time) records the time of the file's last metadata change. It is updated when the inode is modified: when the permission bits are changed, when the owner is changed, when the link count is changed, when the file's size changes, when the file is renamed, and when the file's content is modified. Thectimeis therefore a superset of themtime: everymtimeupdate also updates thectime, but thectimecan be updated without themtime.
ctime but not mtime is
chmod: changing a file's permission
bits modifies the inode (the mode field) but does
not touch the file's contents, so the
ctime is updated and the
mtime is not. The same is true for
chown (changing the owner),
chgrp (changing the group), and
link (adding a hard link, which
updates the link count). The pattern is that
operations that change the file's metadata but not
its contents update only the ctime.
Question 5
What is a sparse file? How does its logical size differ from its allocated size, and how does the file system represent the holes? Give two examples of applications in which sparse files are useful.
Show answer
st_size) includes the holes and can be
very large; its allocated size
(st_blocks × 512) includes only the
blocks that actually hold data and can be very
small. For example, a file with a logical size of
1 GB and two bytes of data at the start and the end
can have an allocated size of only a few kilobytes,
because the 1 GB of zeros between the two bytes is
a hole. Sparse files are created when a program
seeks past the end of a file and writes data there
(as with lseek followed by
write), or when a program explicitly
creates a hole with ftruncate or
fallocate. They are preserved by
programs that are aware of holes (such as
cp --sparse=always and
rsync --sparse) and lost by programs
that read and write every byte (such as
cp without the option). Two examples
of applications: virtual machine disk
images, where the image's logical size is
the size of the virtual disk but only the space the
guest has actually written is allocated — this
allows a VM to have a large disk without consuming
the full space, and it allows snapshots to be
stored efficiently; and scientific and
database files, where a file is
pre-allocated to a maximum size but most of the
space is not yet used — the file system allocates
blocks only as the data grows, and the file's
logical size is the maximum while its allocated
size is the actual usage. Both cases benefit from
the same property: the file appears to be much
larger than the space it actually consumes, which
allows programs to use simple, fixed-size file
layouts without paying the cost of allocating the
full space upfront.
08 Unit discussion
Discussion prompt
File metadata is one of those things that nobody
thinks about until it goes wrong. When a file's mtime
is wrong, make rebuilds things it should
not; when a file's ctime is wrong, an intrusion
detection system misses a compromise; when a file's
permission bits are wrong, a program either cannot
access a file it needs or grants access to a file it
should protect. The metadata is silent, but its
correctness matters.
This lesson has presented the classical metadata
interface: the struct stat structure, its
fields, and the semantics of each. But the interface
is not the whole story. Modern file systems extend the
metadata in many directions: extended attributes allow
arbitrary key-value pairs to be attached to a file;
ACLs extend the classical permission model with
per-user and per-group entries; the statx
interface adds new fields (birthtime, mount ID, file
attributes); and the trend is for each new file system
to add its own metadata. The result is a metadata
landscape that is more complex and less portable than
the classical interface suggests.
Is this a problem? Should file metadata be more standardised, so that a program that reads metadata on one system can interpret it on another? Or is the diversity appropriate, since different file systems have different capabilities and different workloads need different metadata? Consider the cloud storage services that have displaced traditional file systems for many workloads: they typically have a much smaller metadata model (a key, a value, and a few attributes), and they sacrifice much of the classical model's richness for scalability and simplicity. Which model is the right one for the next decade?
Post your response to the Unit 10 discussion forum and reply to at least two classmates. Supports CLO1, CLO5, and CLO6.
09 Practice exercise
This ungraded exercise consolidates the lesson and prepares you for Lesson 10.3. Aim to spend 60–90 minutes on it.
-
Interpret a
struct stat. For the following file metadata, interpret each field, identify the file's type, compute its access permissions, and state whether the file has holes.
Then state the file's logical size and its allocated size in bytes, and explain the difference.st_dev = 0x0801 st_ino = 524291 st_mode = 0x81A4 st_nlink = 1 st_uid = 1000 st_gid = 100 st_size = 1073741825 st_blocks = 8 -
Compare
stat,lstat, andfstat. Create a symbolic link to a file and compare the metadata returned by the three calls. What is the link'sst_size, and what does it represent? What is the target'sst_size? Write the answers in your own words. -
Trace the permission check.
A file has the following metadata:
st_uid = 1000,st_gid = 100,st_mode = 0640. For each of the following processes, trace the permission check and state whether the process can read the file, write to the file, or execute the file:- Process runs as UID 1000, GID 100.
- Process runs as UID 1000, GID 200 (not a member of 100).
- Process runs as UID 2000, GID 100.
- Process runs as UID 2000, GID 200.
- Process runs as UID 0, GID 0.
-
Observe the timestamps.
Create a file, then perform a sequence of operations:
write to it, read from it, change its permissions,
change its owner (if you have permission), and rename
it. After each operation, run
statand record which timestamps changed. Present the results in a table and explain each change in terms of what the operation does to the file's contents and metadata. -
Compute the disk usage of sparse files.
Create a sparse file of 1 GB with two bytes of data
(one at the start and one at the end), then compute
its logical size and its allocated size. Copy it with
cp,cp --sparse=always, andcp --sparse=never, and compare the disk usage of the three copies. Explain the differences.du -h --apparent-size sparse.bin # logical size du -h sparse.bin # allocated size ls -lh sparse.bin # human-readable logical size stat -c 'size=%s blocks=%b' sparse.bin -
Compare the metadata of files on different
file systems.
If your system has multiple file systems, create
files on each and compare their metadata. Do all file
systems support
st_birthtim? Do they support extended attributes? Do they support ACLs? The answers depend on the file system's features.# Find the file systems df -T # Create a file on each and check its metadata for dir in /tmp /var/tmp /home; do echo "=== $dir ===" touch $dir/test-$$.txt stat $dir/test-$$.txt | head -8 rm $dir/test-$$.txt done -
Write a program that reads metadata.
Write a C program that takes a path as its argument,
calls
lstat, and prints the file's type, size, mode (in symbolic form), owner, group, link count, and three timestamps in human-readable form. Use the symbolic constants (S_ISREG,S_ISDIR,S_ISLNK, and so on) to determine the type, and usestrftimeto format the timestamps.
Test the program on a regular file, a directory, a symbolic link, a device file, and a FIFO.#include <sys/stat.h> #include <stdio.h> #include <time.h> int main(int argc, char **argv) { struct stat st; if (lstat(argv[1], &st) == -1) { perror("lstat"); return 1; } /* determine the type */ if (S_ISREG(st.st_mode)) printf("regular file\n"); else if (S_ISDIR(st.st_mode)) printf("directory\n"); else if (S_ISLNK(st.st_mode)) printf("symbolic link\n"); /* ... other types ... */ /* print the mode in symbolic form */ /* print the timestamps */ /* ... */ return 0; } - Reflect. In your own words, explain the difference between a file's logical size and its allocated size, and give one situation in which the difference is important. Then describe one situation in your own experience where a file's metadata was important — perhaps the modification time was used by a build system, or the permission bits determined whether a program could run, or the access time was used by a backup tool. What was the situation, and what would have gone wrong if the metadata had been incorrect?
10 Summary
- File metadata is the information a file system maintains about a file, excluding its contents. It includes the file's identity, type, ownership, permissions, size, timestamps, and other attributes.
-
The metadata is exposed to user space through the
statfamily of system calls, which fill in astruct statstructure. The three calls —stat,lstat, andfstat— differ in how they interpret the argument:statfollows symbolic links,lstatdoes not, andfstatoperates on an open file descriptor. -
A file's identity is the pair
(st_dev, st_ino): the device and the inode number within that device. The pair is used to uniquely identify a file, because inode numbers are meaningful only within a single file system. -
The link count (
st_nlink) is the number of hard links to the file. The inode is freed only when the link count reaches zero and no process has the file open. -
The mode bits (
st_mode) encode the file's type and its permission bits. The types are regular file, directory, symbolic link, character device, block device, FIFO, and socket. The permission bits are the owner, group, and other triplets, plus the set-user-ID, set-group-ID, and sticky bits. - The permission-check algorithm checks the effective UID and GID against the file's owner and group. Root bypasses the checks (with two exceptions). Otherwise, the first matching category — owner, group, or other — determines the permissions used; the kernel applies exactly one set of bits, not all three.
-
The three timestamps record different
events:
atimeis the time of the last read,mtimeis the time of the last content modification, andctimeis the time of the last metadata change (including content changes). Thebirthtime(creation time) is available on modern file systems throughstatx. -
Linux supports three policies for updating
atime:strictatime(every read),relatime(the default; only when the current atime is older than mtime or ctime, or after a day), andnoatime(never). The choice balances performance and the usefulness of the field. -
A file's logical size
(
st_size) is the number of bytes in the file; its allocated size (st_blocks × 512) is the space it occupies on disk. The two can differ because of block alignment or because the file is sparse — a file with holes, whose logical size includes regions that contain no data and are not allocated on disk. -
Extended attributes (xattrs) are
named key-value pairs associated with a file, in
namespaces such as
user.,trusted.,security., andsystem.. They are used for ACLs, security labels, file capabilities, file-system bookkeeping, and application metadata. -
Access control lists (ACLs) extend
the classical permission model with entries for
specific users and groups. They are stored in the
system.posix_acl_accessextended attribute, and they interact with the classical permission bits in ways that are worth understanding. - The metadata lives in the file's inode (or its equivalent in modern file systems). The inode's structure and lifetime are the subject of Lesson 10.3; the strategies for laying out the metadata and for keeping it consistent in the face of crashes are the subject of Lessons 10.5 and 10.6.
Looking ahead
This lesson presented file metadata: the
stat interface, the fields of the
structure, the permission-check algorithm, the
timestamps and their semantics, the size and block
counts, and the extended attributes and ACLs that
modern file systems support. It referred several times
to the inode — the file system's own
structure that holds the metadata — but did not
present it in detail. Lesson 10.3 presents the inode
itself: its structure, its fields, its role in path
resolution, its lifetime, and the operations that
manipulate it. The lesson examines the classical Unix
inode, the changes introduced by the Berkeley Fast
File System, and the modern inode structures used in
ext4, XFS, Btrfs, and ZFS. It closes Unit 10's
treatment of the file's identity and metadata, and it
prepares the way for the discussion of directory
organization (Lesson 10.4) and file allocation
(Lesson 10.5).
11 References
- Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 13, Section 13.1, and Chapter 14, Section 14.2.
-
Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C.
Operating Systems: Three Easy Pieces, Chapters 39
and 40. Available online at
ostep.org. - Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 4, Sections 4.1 and 4.3.
-
Kerrisk, M. The Linux Programming Interface.
No Starch Press, 2010. Chapters 15 and 16. The
definitive reference for the
statstructure, its fields, and Linux's extended attributes. - Bryant, R. E., & O'Hallaron, D. R. Computer Systems: A Programmer's Perspective, 3rd ed., Chapter 10, System-Level I/O.
- McKusick, M. K., Joy, W. N., Leffler, S. J., & Fabry, R. S. A Fast File System for UNIX. ACM Transactions on Computer Systems, 1984. The paper that refined the inode structure and the metadata layout.
- Ritchie, D. M., & Thompson, K. The UNIX Time-Sharing System. Communications of the ACM, 1974. The original paper that established the Unix file and metadata model.
-
IEEE Std 1003.1-2017 (POSIX.1-2017). The specification
of the
struct statfields and thestatfamily of system calls. -
Linux man-pages:
man 2 stat,man 2 lstat,man 2 fstat,man 2 statx,man 2 chmod,man 2 chown,man 2 utime,man 2 utimensat,man 2 truncate,man 7 xattr,man 5 acl,man 5 proc,man 1 stat,man 1 getfattr,man 1 setfattr,man 1 getfacl,man 1 setfacl.