COMP 3150Operating Systems

Module 1 · Unit 1 · Lesson 4 of 5

System Calls

The mechanism by which a program requests a service from the kernel — and the hardware-supported boundary that separates user space from kernel space.

01 Introduction

Lesson 1.3 catalogued the services an operating system provides. This lesson examines the interface through which those services are requested: the system call. System calls are the only legitimate way for a user-space program to ask the kernel to do something on its behalf.

The system call is more than a function call. It involves a deliberate, hardware-mediated transition from the unprivileged mode in which applications run to the privileged mode in which the kernel runs. This transition — sometimes called a mode switch, trap, or context switch in the loose sense — is central to how operating systems enforce protection and isolation.

Understanding system calls is essential for two reasons. First, they are the mechanism through which every program exercises OS services, so knowing them is knowing the OS's actual interface. Second, system-call overhead is a real performance factor, and tools like strace make that overhead visible.

02 Learning objectives

By the end of this lesson you will be able to

  • Explain system calls.
  • Distinguish APIs from system calls.
  • Explain user mode and kernel mode.
  • Explain traps and interrupts.
  • Use Linux tools to observe system calls.

This lesson contributes to U1.4, U1.3, and course learning outcome CLO1. It also prepares the ground for CLO6.

03 Required reading

  • Operating System Concepts — Chapter 2, Section 2.3, System Calls.
  • OSTEP — Chapter 4, The Abstraction: The Process. The fork, exec, and wait examples are direct illustrations of system-call programming.
  • Linux man pages: man 2 syscall, man 2 read, man 2 write, man 1 strace.

04 Core content

4.1 What a system call is

A system call is a request from a user-space program to the operating-system kernel to perform a privileged operation on the program's behalf. The operation might be reading a file, creating a process, allocating memory, sending a packet, or querying the current time. Whatever the operation, it requires the kernel's authority.

A user-space program cannot perform these operations itself for a simple reason: the hardware does not allow it. The CPU has at least two privilege levels — user mode and kernel mode — and instructions that manipulate device registers, alter page tables, or configure interrupts are available only in kernel mode. To reach kernel mode, the program must ask the kernel to switch for it. The system call is that request.

4.2 User mode and kernel mode

The distinction between user mode and kernel mode is enforced by the CPU. Modern processors have at least two privilege levels, sometimes called rings:

Privilege levels on a typical x86-64 processor
Level Name What runs there Privileged instructions
Ring 0 Kernel mode The kernel, its device drivers, loadable modules Available
Ring 1 Historically for device drivers; rarely used on modern systems Available
Ring 2 Historically for device drivers; rarely used on modern systems Available
Ring 3 User mode Applications, libraries, services Blocked — causes a trap if attempted

In practice, almost everything runs either in ring 0 or ring 3. The intermediate rings exist for historical reasons and are not meaningfully used by mainstream operating systems today.

Two consequences follow from this design:

  • A bug in user-space code can only damage that process; it cannot corrupt the kernel.
  • Kernel code is trusted absolutely. A bug in the kernel can bring down the entire machine or compromise security.

4.3 APIs versus system calls

Most programs never issue a system call directly. Instead, they call a function in a library — usually the C standard library, libc — which issues the system call internally. The interface the programmer sees is the API; the interface the kernel sees is the system call.

The two are not the same. Consider two very different cases:

One-to-one: read()
The libc wrapper for read is essentially a thin veneer: it moves the arguments into the correct registers and executes the trap instruction. One API call maps to one system call.
Many-to-many: printf()
The printf API can be implemented with very different system-call patterns depending on buffering. On the first call with buffered output, printf may issue no system call at all. Later, when the buffer fills or is flushed, it issues a single write. A program that calls printf a thousand times may issue only a handful of system calls.

Why the distinction matters

The C library provides a portable, convenient interface that abstracts over kernel differences. The same program can compile and run on Linux, macOS, and FreeBSD, even though the underlying system calls differ between them. Knowing the difference also matters for performance: a library call that does not touch the kernel is orders of magnitude cheaper than one that does.

4.4 Traps and interrupts

The mechanism that switches the CPU from user mode to kernel mode is called a trap (or, on some architectures, a software interrupt or supervisor call). A trap is a synchronous event caused by the currently executing instruction. The syscall instruction on x86-64 is the canonical example: executing it transfers control to a fixed entry point in the kernel and raises the privilege level to ring 0.

A hardware interrupt differs in one crucial respect: it is asynchronous. It is caused by an external device — a timer, a disk controller, a network card — and it can occur at any time, not just at a specific instruction. Interrupts also enter the kernel and raise the privilege level, but they are not initiated by the running program.

Comparing traps and interrupts
Characteristic Trap (software) Interrupt (hardware)
Cause Executing a specific instruction An external device signals the CPU
Timing Synchronous — happens at a known instruction Asynchronous — can happen at any time
Initiated by The running program A device
Typical use System calls, faults, breakpoints Timer ticks, I/O completion, network packets
Handler entry Kernel trap handler, dispatched by syscall number Kernel interrupt handler, dispatched by IRQ number

Both mechanisms share the essential property of switching the CPU into kernel mode, saving the interrupted context, and vectoring to a handler. Both return to user mode when the handler completes. The distinction between them is one of origin and timing, not of privilege level.

4.5 The system-call sequence in detail

It is worth tracing a single system call end to end. Suppose a program calls read(fd, buf, 4096).

The system-call sequence from user space to kernel space and back A sequence diagram. The program calls the libc read wrapper, which places the syscall number and arguments in registers and executes the syscall instruction. The CPU traps into kernel mode, the kernel dispatches to the read handler, performs the operation, stores the return value, and executes a return-from-trap instruction. The program resumes with the result. USER SPACE KERNEL SPACE program calls read(fd, buf, 4096) libc wrapper loads syscall number + args into registers syscall instruction trap · raise privilege · vector to entry kernel entry point save registers · dispatch by syscall number sys_read handler validate fd · check buffer · perform I/O (if data not ready) block · schedule another process store return value bytes read · or -errno return-from-trap drop privilege · restore context program resumes returns the byte count
Figure 1.4 — The system-call sequence. The mode switch is hardware-mediated; the dispatch and handler logic are kernel software. On return, the process continues as if a normal function call had completed.

The numbered steps in the figure correspond to what actually happens on a modern Linux system. Three details are worth emphasising:

  • The syscall number matters. The kernel does not know which system call was requested from the trap instruction alone. The caller places the syscall number in a designated register (for example, rax on x86-64), and the entry point uses it to index a table of handlers.
  • Arguments pass in registers, not on the stack. Stack switching is part of the trap sequence, so arguments are passed via specific registers. On x86-64 Linux the first six arguments go in rdi, rsi, rdx, r10, r8, and r9.
  • Return values encode errors. Linux system calls return a non-negative value on success; on error they return -errno. The libc wrapper translates this into the familiar -1 return plus a global errno variable.

4.6 Categories of system call

System calls group naturally by the service they provide. The categories mirror Lesson 1.3.

Representative Linux system calls by category
Category Representative calls
Process control fork, execve, exit, wait4, kill, clone
File management open, close, read, write, lseek, stat, unlink
Device management ioctl, mmap, read, write
Information maintenance getpid, alarm, sleep, time, sysinfo
Communications pipe, socket, connect, sendto, recvfrom
Protection chmod, chown, umask, setuid, capset

4.7 Observing system calls on Linux

Linux provides several tools for observing the system-call interface at runtime. Two of them are essential.

strace
Traces the system calls made by a program. Each call is printed with its arguments and its return value. This is the single most useful tool for understanding what a program actually does at the kernel boundary.
ltrace
Traces library calls rather than system calls. Useful for seeing what happens above the syscall boundary — for example, how many printf calls a program makes before a single write reaches the kernel.

Two strace invocations to remember: 

# Summary of system calls by count and time
strace -c ./myprogram

# Full trace with timestamps
strace -tt -T ./myprogram

The -c flag produces a summary table; the -tt and -T flags in the second command add timestamps and per-call durations. Reading these outputs is a skill you will use throughout the course.

05 Worked example

Read and interpret the output of a real system-call trace.

Trace of a small file-reading program

Command:

strace -tt -e trace=openat,read,write,close cat /etc/hostname

Sample output (abbreviated):

12:30:41.112 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
12:30:41.113 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
12:30:41.115 openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3
12:30:41.115 read(3, "myhost.example.local\n", 131072) = 21
12:30:41.116 write(1, "myhost.example.local\n", 21) = 21
12:30:41.116 read(3, "", 131072) = 0
12:30:41.116 close(3) = 0

Reading the trace line by line:

  1. openat(AT_FDCWD, "/etc/ld.so.cache", ...) = 3 — the dynamic linker opens the library cache. The return value 3 is a file descriptor.
  2. The second openat loads libc. Again, a new file descriptor is returned.
  3. The third openat opens the requested file, /etc/hostname, also returning descriptor 3 (the earlier descriptors were closed by the linker once they were no longer needed).
  4. read(3, "myhost.example.local\n", 131072) = 21 — the kernel returns 21 bytes into the buffer, even though the buffer is 131072 bytes long. This is normal: read returns as soon as any data is available.
  5. write(1, "myhost.example.local\n", 21) = 21 — the program writes the same 21 bytes to descriptor 1, standard output.
  6. The second read returns 0, which means end of file.
  7. close(3) = 0 — the descriptor is released.

What the trace reveals

Running cat is more than reading a file. The dynamic linker runs first, several files are opened, and the program issues exactly five of the traced system calls in its own right. This kind of insight is impossible to get from source code alone when libraries are involved.

06 Interactive activity

Activity · observe, compare, explain

Perform the following in your Linux VM.

  1. Establish a baseline. Run strace -c /bin/true. Note the total number of system calls and the top three by count.
  2. Compare with a real command. Run strace -c ls /usr/bin. How many more system calls were issued? Which category dominates?
  3. See buffering. Compile a program that calls printf in a loop 1000 times, each on a separate line, writing to a file. Run it under strace -c. Count the write calls. How does the count compare with 1000?
  4. Trace a block. Run time cat /dev/zero | head -c 100000000 > /dev/null. Then run it again under strace -c. Note any difference in wall-clock time.
  5. Reflect. Which system call dominated in each trace? What does that tell you about the workload?

07 Knowledge check

Check your understanding

Question 1

A program calls printf("hello\n") once and then exits. How many write system calls will it issue? Explain.

Show answer
Exactly one — assuming standard output is a file or pipe and not a terminal. The printf call writes into a library buffer; no system call occurs yet. When the program exits, the library's atexit handlers flush the buffer with a single write. If output is a terminal, libc uses line buffering, so printf issues a write immediately on seeing the newline — still one call, but for a different reason.

Question 2

Explain the difference between a trap and an interrupt. Give one example of each.

Show answer
A trap is a synchronous, software-initiated event caused by executing a specific instruction. The syscall instruction on x86-64, which requests a kernel service from a user program, is a trap. An interrupt is an asynchronous, hardware- initiated event caused by an external device. A timer tick that preempts the running process is an interrupt.

Question 3

Why does the operating system place the system-call number in a register rather than passing it as an ordinary function argument?

Show answer
Because the trap itself must be a single, fixed instruction. The kernel's entry point is at a well-known address; it has no access to the caller's stack yet. It must learn which system call is being requested from information that has already been placed somewhere the CPU can read after raising the privilege level — that is, in a register. On x86-64 Linux, rax carries the system-call number and the remaining registers carry the arguments.

08 Unit discussion

Discussion prompt

System-call overhead is real: on a typical modern CPU, a system call costs hundreds to thousands of cycles. Applications in performance-sensitive domains — databases, network servers, trading systems — go to considerable lengths to reduce it, using mechanisms such as io_uring, mmap, or kernel bypass.

Identify one application domain where reducing system-call overhead is worth substantial engineering effort, and one where it is not. Justify your choices with reference to the workload characteristics.

Post your response to the Unit 1 discussion forum and reply to at least two classmates. Supports CLO1 and CLO6.

09 Practice exercise

  1. Read the wrapper. Find the libc source for read. On a typical Linux system it is implemented in assembly, in a file such as sysdeps/unix/sysv/linux/x86_64/syscall.S. How many instructions does it take?
  2. Estimate syscall overhead. Write a small C program that issues getpid() one million times and measures the elapsed time. On a modern Linux system a getpid call is a few hundred nanoseconds. What does that tell you about the cost of a typical system call?
  3. Compare buffered and unbuffered I/O. Write a program that writes 1 MB to a file using write() in 1-byte increments. Time it. Then write the same amount using fwrite. Time it. Compare the two, and explain the difference in terms of system calls.
  4. Trace a fork. Write a program that calls fork(). Trace it with strace -f -e trace=clone,execve,wait4. What underlying system call does fork use on Linux?

10 Summary

  • A system call is a request from a user-space program to the kernel to perform a privileged operation.
  • The distinction between user mode and kernel mode is enforced by the CPU and underpins all OS protection.
  • Library APIs are not the same as system calls: one API call may map to zero, one, or many system calls.
  • A trap is a synchronous software-initiated mode switch; an interrupt is an asynchronous hardware-initiated one. Both enter the kernel at privileged level.
  • The system-call sequence involves saving the user context, dispatching on the syscall number, performing the operation, and returning via a return-from-trap instruction.
  • strace and ltrace make the system-call interface observable, and reading their output is a core skill for the rest of the course.

Looking ahead

Lesson 1.5 considers how the kernel that handles these system calls is organised internally — the major architectural alternatives and their trade-offs.

11 References

  • Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, Chapter 2.
  • Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapter 4.
  • Love, R. Linux System Programming, Chapter 1.
  • Linux man-pages: man 2 syscall, man 1 strace.
👤