Kernel Threads

01 Introduction

Lesson 4.2 examined the user-thread model and identified two fundamental limitations. First, a blocking system call in any one user thread stalls the entire process, because the kernel suspends the process and the library cannot run. Second, user threads of one process cannot run on multiple CPU cores at once, because the kernel sees only one schedulable entity.

Both limitations have the same root cause: the kernel does not know the threads exist. The natural fix is to change that — to make the kernel aware of each thread and let it schedule them independently. This is the kernel-thread model, and it is what modern general-purpose operating systems — Linux, Windows, macOS, and the BSDs — actually implement.

Kernel threads solve both fundamental problems. If one thread blocks, the kernel simply schedules another. If the machine has multiple cores, the kernel can dispatch different threads to different cores and get true parallelism. The cost is that thread operations now require kernel involvement: creating and switching threads is more expensive than in the user-thread model, because the kernel must allocate resources and take a trap. The trade-off is worth it for the vast majority of workloads.

This lesson defines what a kernel thread is, examines how the kernel represents and schedules threads, presents the four classical threading models that combine user and kernel support, analyses the costs, and closes with the concrete Linux implementation. It also distinguishes kernel-managed user threads from Linux's own internal kernel threads — a related but distinct concept.

02 Learning objectives

This lesson contributes to U4.3 and to course learning outcomes CLO2 and CLO8.

03 Required reading

• Operating System Concepts — Chapter 4, Section 4.3, Multithreading Models, and Section 4.5, Multicore Programming. The classical taxonomy of the four threading models and the multicore considerations.
• OSTEP — Chapter 26, the sections on kernel-level threading and the threading APIs. OSTEP's framing of the trade-offs is concise.
• Love, R. — Linux Kernel Development, 3rd ed., Chapter 3 (Process Management). The section on task_struct and thread creation via clone.
• Drepper, U., & Molnar, I. — The Native POSIX Thread Library for Linux. Red Hat, 2003. The design document for NPTL, the implementation Linux uses today.
• Linux man pages: man 2 clone, man 7 pthreads, man 3 pthread_create, man 5 proc (for /proc/<pid>/task/), man 1 ps (the -L and -T options).

04 Core content

4.1 What a kernel thread is

A kernel-level thread — often just called a "kernel thread" in this context — is a thread that the kernel creates, schedules, and manages directly. The kernel is aware of the thread as an independent schedulable entity, not as part of some larger process that it schedules as a unit. Each kernel thread has:

• Its own kernel-side task structure. On Linux this is a task_struct — the same structure that represents a process. A thread is, from the kernel's point of view, a task that happens to share some resources with other tasks.
• Its own kernel stack. When the thread makes a system call or handles an interrupt, it uses its own kernel-mode stack, separate from every other thread's. On Linux the kernel stack is typically 8 or 16 KB, allocated when the thread is created.
• Its own register state. The program counter, general-purpose registers, and status flags are saved and restored by the kernel on each context switch, just as they are for a process.
• Its own scheduling record. Priority, virtual runtime, and the other scheduling fields are per-thread. The kernel's scheduler dispatches threads, not processes.

What a kernel thread does not have, in general, is a private address space. Threads of the same process share one address space; the kernel's memory-management structure (mm_struct on Linux) is shared by all threads of that process. This is what makes a kernel thread cheaper than a kernel process: the kernel does not need to duplicate the page table or the file descriptor table.

4.2 The kernel's view: tasks and thread groups

On Linux, the kernel represents every schedulable entity with the same data structure, struct task_struct, introduced in Lesson 3.3. A process and a thread are both instances of task_struct; the difference is in which fields they share.

Specifically, threads of the same process share:

• The address space — the mm_struct is the same.
• The file descriptor table — the files_struct is the same.
• The signal handlers — the sighand_struct is the same.
• The credentials — the cred structure is shared.
• The thread group ID (TGID), which is what user space calls the "process ID" of the group.

And each thread has its own:

• Kernel stack — a separate region of kernel memory.
• User-space stack — a separate region of the shared address space.
• Program counter and register set.
• Thread ID (TID) — distinct from the PID of the group.
• Scheduling parameters — priority, CFS virtual runtime, time slice.

This design is sometimes described as "Linux has no concept of a thread" — because both processes and threads are tasks. In fact Linux has a very precise concept of a thread: it is a task that shares a set of resources with other tasks in the same thread group. The distinction between processes and threads is not about a separate data structure, but about which resources are shared.

4.3 How kernel threads solve the blocking problem

Return to the scenario from Lesson 4.2. Thread 1 calls read on a slow device. The kernel's read handler determines that the data is not immediately available and marks the calling task as Blocked, moving it off the run queue.

With kernel threads, this is where the story differs. The kernel has moved one task to the Blocked state. The other threads of the same process are separate tasks, each with its own state in the process table. The scheduler looks at the ready queue and finds those other threads — runnable, waiting for a CPU. It dispatches one of them.

Thread 2 now runs. It has no idea that Thread 1 is blocked in the kernel; it just sees the shared memory as it was, and continues executing. When the disk eventually responds, the kernel wakes Thread 1 (moving it from Blocked to Ready) and it will be scheduled again in due course.

From the application's point of view, the process has simply kept running — one thread waiting on I/O, the others continuing their work. This is what "concurrency" was supposed to mean all along, and it is what user threads could not deliver.

4.4 How kernel threads solve the multicore problem

The multicore problem is solved by the same change in the kernel's model. Because each thread is an independent schedulable entity, the kernel can dispatch different threads of the same process onto different CPU cores simultaneously. Thread 1 runs on core 0 while Thread 2 runs on core 1 while Thread 3 runs on core 2 — all of them executing code from the same address space at the same time.

This is what makes multithreaded programs scale. A program that was limited to one core as a single-threaded process becomes capable of using all available cores as a multithreaded one. The performance gain can be large — a four-thread program on a four-core machine can complete in roughly a quarter of the time if the workload is parallelisable.

The kernel's job in supporting this is straightforward: its scheduler must be able to run more than one task at once, and its per-CPU run queues must be able to hold tasks from the same thread group. Modern Linux does exactly this with per-CPU run queues and a global load balancer. Lesson 4.4 examines the details of multicore scheduling.

4.5 The four threading models

Once we have both user-level threads and kernel-level threads, the question becomes how to combine them. The classical literature identifies four models, distinguished by the relationship between user threads (as the application sees them) and kernel threads (as the kernel schedules them).

Many-to-one

Many user threads multiplexed onto a single kernel thread. This is the user-thread model of Lesson 4.2, viewed from the perspective of a system that also supports kernel threads. Fastest switching, but no parallelism and the blocking problem remains.

One-to-one

Each user thread has its own kernel thread. The kernel schedules each thread independently. True parallelism, blocking is per-thread. More expensive per thread. This is the model Linux, Windows, and macOS use.

Many-to-many

Many user threads multiplexed onto a smaller or equal number of kernel threads. Combines cheap user-level switching with partial parallelism and blocking tolerance. Complex to implement correctly. Solaris implemented this; most modern systems have moved away from it.

Two-level

A variant of many-to-many that also allows a user thread to be bound to a specific kernel thread. Historical; largely superseded by one-to-one in practice.

4.6 How many kernel threads?

A subtle but important design question: if the kernel is going to schedule threads, how many kernel threads should exist for a given number of user-level tasks? The answer differs by model and by system.

• Many-to-one. Exactly one kernel thread per process, regardless of how many user threads the application creates. The application can create a million logical tasks but only one of them runs at a time.
• One-to-one. One kernel thread per user thread, created on demand. The kernel schedules each thread independently. There is no multiplexing; the number of kernel threads equals the number of user threads.
• Many-to-many. The library chooses a number of kernel threads — usually equal to the number of CPU cores, or a small multiple of that — and multiplexes its user threads onto them. This keeps the number of kernel-scheduled entities bounded while still exploiting all cores.

On Linux, the one-to-one model means that an application that creates 10,000 pthreads creates 10,000 kernel tasks. This is feasible because each kernel task is fairly small, but it does consume memory (kernel stack plus task_struct) and the scheduler's run queue becomes large. Language runtimes that want very large concurrency (Go, Erlang) use the many-to-many model internally, precisely so that the number of kernel- scheduled entities stays bounded even when the number of logical tasks is huge.

4.7 Costs of kernel threads

Kernel threads solve the two problems of the user-thread model, but they are not free. Three costs matter.

4.8 Linux's implementation

Linux uses the one-to-one model, implemented through two mechanisms that work together: the clone system call and the Native POSIX Thread Library (NPTL).

clone. Linux creates both processes and threads with a single system call, clone. The difference is which resources the new task shares with its creator. The clone call takes a set of flags that control sharing:

/* A Linux thread is created with clone and a set of sharing flags */
clone(thread_fn, stack_top,
      CLONE_VM      |   /* share address space */
      CLONE_FS      |   /* share filesystem info (cwd, root) */
      CLONE_FILES   |   /* share file descriptor table */
      CLONE_SIGHAND |   /* share signal handlers */
      CLONE_THREAD  |   /* same thread group (same PID to user space) */
      CLONE_SYSVSEM,    /* share System V semaphore undo */
      arg);

The same clone call with different flags — omitting the sharing flags — creates a traditional process. This is why we said that Linux has no separate "thread" concept: it has one task creation primitive, parameterised by what is shared.

NPTL. The Native POSIX Thread Library is the userspace side of the same model. It was introduced in glibc 2.3 (2002) and integrated with the 2.6 kernel (2003). Before NPTL, Linux used LinuxThreads, which had significant problems — separate PIDs for threads, expensive signal-based synchronization, and incorrect POSIX semantics in several corner cases. NPTL solved these by using clone directly and by taking advantage of the kernel's futex mechanism for fast user-space synchronization (covered in Unit 6).

When you call pthread_create on a modern Linux system, here is what happens: NPTL calls clone with the sharing flags, the kernel creates a new task that shares the address space, file descriptors, and signal handlers of the caller, and a new pthread handle is returned to the caller. The new kernel task appears in /proc/<pid>/task/<tid> and is scheduled independently by the kernel's CFS scheduler.

4.9 Linux kernel threads (a distinct concept)

There is a naming collision worth clarifying. In Linux, the term "kernel thread" is also used for a different concept: a task that runs entirely in kernel mode and has no user-space component at all. These are the threads that handle background work inside the kernel — reclaiming memory, flushing dirty pages, running deferred work, migrating tasks between cores.

Examples include:

• kswapd — the kernel's page reclaim daemon. It wakes when memory is low and scans pages for reclaim, running in the background so that user processes are not stalled by memory pressure.
• kworker — the worker pool for deferred kernel work. File-system writeback, network cleanup, and many other tasks are executed by kernel workers.
• migration/N — a per-CPU kernel thread that migrates tasks between CPUs when the load balancer decides to rebalance.
• kthreadd — the parent of all kernel threads. It is PID 2 on a Linux system.

These tasks are visible in ps -ef with names in square brackets and PPID 2, and they can be scheduled by the kernel like any other task. But they are not the same as a user-created thread: they have no user-space stack, no user address space (they borrow the previous task's on a switch, which is why they appear with mm == NULL and use the concept of an "active_mm").

For the rest of this course, when we say "kernel thread" without qualification, we mean the first concept: a user-space thread that the kernel schedules independently. We will use "kernel daemon" or "kernel-internal thread" for the second.

4.10 When kernel threads are the right choice

Kernel threads are the default on modern general-purpose systems, and for most workloads they are the correct choice. But they are not universally better than user threads, and there are workloads where the user-thread model would be preferable in principle.

• Kernel threads are right when blocking is possible. If any thread in the program might block — on I/O, on a lock, on a pipe, on a network — kernel threads give the other threads a way to keep running. This describes almost every real workload.
• Kernel threads are right when parallelism is desired. Any program that wants to use more than one CPU core needs kernel threads (or processes). User threads cannot deliver parallelism on a single-process basis.
• User threads can be better when neither applies. A program that is CPU-bound, runs entirely on one core, and never blocks can use user threads to get cheap cooperative multitasking. This is rare in practice, but the pattern still appears in embedded systems and in specialised runtimes.
• Hybrid models can win when both apply but neither dominates. A runtime that knows its own workload (Go, Erlang) can use a many-to-many model to get cheap user-level switching when the workload is cooperative and kernel-level parallelism when it is not.

The history of the last twenty years is a shift toward kernel-thread models for the OS itself, and toward user-level task models inside language runtimes — with the runtime managing the complex interaction between the two. In both cases, the underlying primitive is the kernel thread; the difference is who schedules it.

05 Worked example

To make the kernel-thread model concrete, look at the same blocking scenario from Lesson 4.2 and see how kernel threads change the outcome, then measure the difference in cost.

A: 0
B: 0
A: 1
A: about to read from stdin (will block)
# ... program hangs until input is provided ...
A: got input
B: 1
B: 2
... continues

#include <stdio.h>
#include <unistd.h>
#include <pthread.h>

void *worker_a(void *arg) {
    (void)arg;
    for (int i = 0; i < 5; i++) {
        printf("A: %d\n", i);
        if (i == 1) {
            printf("A: about to read from stdin (will block)\n");
            char buf[16];
            read(0, buf, sizeof(buf) - 1);
            printf("A: got input\n");
        }
    }
    return NULL;
}

void *worker_b(void *arg) {
    (void)arg;
    for (int i = 0; i < 5; i++) {
        printf("B: %d\n", i);
        /* Busy work to keep the thread active */
        for (volatile long j = 0; j < 10000000L; j++) ;
    }
    return NULL;
}

int main(void) {
    pthread_t ta, tb;
    pthread_create(&ta, NULL, worker_a, NULL);
    pthread_create(&tb, NULL, worker_b, NULL);
    pthread_join(ta, NULL);
    pthread_join(tb, NULL);
    return 0;
}

gcc -Wall -Wextra -g -O2 -pthread -o kthreads kthreads.c
./kthreads

A: 0
B: 0
A: 1
A: about to read from stdin (will block)
B: 1
B: 2
B: 3
B: 4
# ... and after you press Enter ...
A: got input
A: 2
A: 3
A: 4

ps -L -C kthreads -o pid,tid,stat,comm

Kernel-thread switch (approximate, on a modern x86-64 VM):
    Total wall-clock for 2 million yields:  0.68 s
    Cost per yield (two switches):          ~340 ns
    Cost per switch:                        ~170 ns

User-thread switch (measured in Lesson 4.2):
    Cost per switch:                        ~70 ns

Ratio (kernel/user):                      ~2.4×

PID=$(pgrep kthreads)
ls /proc/$PID/task/
# 1234  1235
# Each of these is a kernel task ID; there are two, one per thread.

cat /proc/$PID/status | grep -E 'Name|Tgid|Pid|Threads'
# Name:    kthreads
# Tgid:    1234
# Pid:     1234
# Threads: 2

cat /proc/$PID/task/1235/status | grep -E 'Name|Tgid|Pid'
# Name:    kthreads
# Tgid:    1234
# Pid:     1235
# ...
# Both tasks share the same Tgid (the thread group ID, which is what
# user space calls the process ID) but have distinct Pids (the kernel's
# per-task ID, which user space calls the thread ID).

06 Interactive activity

07 Knowledge check

08 Unit discussion

09 Practice exercise

10 Summary

• A kernel-level thread is a thread the kernel knows about. The kernel has a task structure, a kernel stack, and scheduling metadata for each one, and it schedules each thread independently.
• Kernel threads solve both fundamental limitations of the user-thread model: a blocking call only blocks the calling thread (the kernel moves on to the others), and threads can run in parallel on multiple CPU cores.
• The four threading models describe how user threads map to kernel threads: many-to-one (user threads only), one-to-one (each user thread has its own kernel thread), many-to-many (multiplexed onto a bounded number of kernel threads), and two-level (many-to-many plus some bound threads).
• Kernel threads cost more than user threads: creating one requires a system call and kernel resources; switching between them requires a trap into the kernel. The cost is typically a few microseconds, versus tens of nanoseconds for a user-thread switch. But the cost is well worth it for the capabilities it buys.
• Linux uses the one-to-one model via two mechanisms: clone (the system call that creates tasks, parameterised by sharing flags) and NPTL (the userspace library that implements POSIX threads on top of clone).
• Linux also has a distinct concept of kernel-internal threads — tasks that run entirely in kernel mode with no user-space component, such as kswapd, kworker, and migration/N. These are kernel daemons, not user threads, and appear in ps -ef with names in square brackets and PPID 2.
• Modern language runtimes such as Go and Erlang implement many-to-many models internally — but the underlying primitive they build on is the OS kernel thread.

11 References

• Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 4, Sections 4.3 and 4.5.
• Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapter 26. Available online at ostep.org.
• Love, R. Linux Kernel Development, 3rd ed., Chapter 3 (Process Management). Addison-Wesley, 2010.
• Drepper, U., & Molnar, I. The Native POSIX Thread Library for Linux. Red Hat, 2003.
• Anderson, T. E., Bershad, B. N., Lazowska, E. D., & Levy, H. M. Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism. ACM TOCS, 1992.
• Linux man-pages: man 2 clone, man 7 pthreads, man 3 pthread_create, man 5 proc, man 1 ps.