User Threads

01 Introduction

Lesson 4.1 established what a thread is and why splitting execution from the process is useful. It left open a design question: where should the thread abstraction be implemented? Two answers exist, and they lead to very different systems.

The first answer is user-level threads: the thread abstraction lives entirely in a library running in the process's own address space. The library provides operations to create, schedule, and switch between threads, and it does all of this without ever asking the kernel for help. From the kernel's point of view, a process with a hundred user-level threads is indistinguishable from a process with one — the kernel sees a single stream of control, and it schedules that stream like any other.

This was the original threading model, developed at Carnegie Mellon and elsewhere in the 1980s, and it has several compelling properties. Switching between user threads is extremely fast because it does not involve a trap into the kernel. The library can implement whatever scheduling policy the application needs. The system requires no kernel support for threads at all, so user threads work on any UNIX-like system.

The model also has one fundamental limitation that eventually pushed modern systems toward kernel-level threads. This lesson examines how user threads work, why they are fast, what they cost, and what the workarounds are — and it prepares the ground for Lesson 4.3, which presents the kernel-level alternative.

02 Learning objectives

This lesson contributes to U4.2 and to course learning outcomes CLO2 and CLO8.

03 Required reading

• Operating System Concepts — Chapter 4, Section 4.3, Multithreading Models. The classical presentation of user threads, kernel threads, and the models that combine them.
• OSTEP — Chapter 26, the discussion of user-level threading and the historical development of threading libraries. OSTEP's framing of the blocking problem is clear and concise.
• Draves, R., Bershad, B., Rashid, R., & Dean, R. — Using Continuations to Implement Thread Management and Communication in Operating Systems. SOSP, 1991. The classical paper on user-level thread implementation techniques, including the scheduler-activation model.
• Linux man pages: man 3 makecontext, man 3 swapcontext, man 3 setcontext. The POSIX ucontext family, which is the modern interface for user-level context switching.
• GNU Pth documentation and the historical documentation for the original Mach C-threads library, both freely available online. These show what a real user-level threading library looks like.

04 Core content

4.1 What a user-level thread is

A user-level thread is a thread that is created, scheduled, and managed entirely by a library running in the process's own address space. The kernel is not involved. It does not allocate any resources for the thread, does not know the thread exists, and does not schedule it.

From the kernel's point of view, a process that uses a user-level thread library looks exactly like a single-threaded process: one program counter, one stack, one kernel-side task. The thread library has created the illusion of concurrency entirely in user space, by rapidly switching among its own threads within the single kernel-visible stream of control.

This is the defining feature of the model. It is also the source of its strengths and its central limitation.

4.2 The library and the thread control block

A user-level threading library maintains its own data structure for each thread, called a thread control block (TCB). The TCB is the user-space analogue of the kernel's process control block from Lesson 3.3, but it is much smaller. It typically holds only what the library needs to switch between threads:

• A saved context. The register values — the program counter, the stack pointer, general-purpose registers, and status flags — from the last time the thread was suspended. This is what allows the thread to resume where it left off.
• A thread-local stack. Each thread must have its own stack, allocated by the library from the process's heap or a separate memory region.
• A state field. The library maintains its own notion of which threads are runnable, which are blocked on what, and which have terminated.
• A thread ID. A library-visible identifier, independent of any kernel-level identifier.
• Scheduling metadata. Priority, runnable time, and whatever else the library's scheduling policy requires.

The library keeps its TCBs in a data structure of its own choosing: a list, a queue, a priority heap, or a red-black tree. This choice belongs to the library and is invisible to the kernel.

4.3 How a user-thread switch works

A user-thread switch is a pure user-space operation: the library saves the outgoing thread's registers into its TCB, restores the incoming thread's registers from its own TCB, and transfers control to the incoming thread's saved program counter. No privileged instruction is executed. No trap into the kernel occurs. No mode switch happens.

On Linux, the modern mechanism for performing this switch is the ucontext family: getcontext, setcontext, makecontext, and swapcontext. These are provided by the C library and implemented in a small amount of assembly. A swapcontext call performs the entire switch and is typically a few dozen instructions.

#include <ucontext.h>

/* A thread control block for a minimal user-thread library */
typedef struct uthread {
    ucontext_t       ctx;       /* saved register state */
    char            *stack;     /* thread-private stack */
    size_t           stack_size;
    int              state;     /* RUNNABLE / BLOCKED / TERMINATED */
    struct uthread  *next;      /* for the library's run queue */
} uthread_t;

/* The user-thread switch: pure user space, no kernel involved. */
void uthread_switch(uthread_t *from, uthread_t *to) {
    from->state = RUNNABLE;
    to->state   = RUNNING;
    swapcontext(&from->ctx, &to->ctx);
}

The absence of any kernel involvement is what makes the switch so cheap. A kernel-mediated thread switch must execute a trap instruction, save state on the kernel stack, run the scheduler, restore state, and execute a return-from-trap — a sequence that costs hundreds to thousands of cycles. A user-space switch is a handful of mov instructions and a ret. In round numbers, a user-thread switch is often 10–50× cheaper than a kernel-thread switch on the same hardware.

4.4 The library schedules, not the kernel

Because the kernel does not know about user threads, it cannot schedule them. The library must. This is a significant responsibility, and it gives the library both an opportunity and a burden.

The opportunity: the library can implement whatever scheduling policy the application needs. A database might use a priority policy tuned for its queries; a parallel-scientific workload might use a work-stealing policy optimised for cache locality; a graph algorithm might use a policy that prioritises breadth-first frontier expansion. A general-purpose kernel could never know enough about the application to make these choices well, but a library can.

The burden: the library must decide when to switch. In a cooperative model — which is what most user-level thread libraries use — the running thread must explicitly yield control by calling a library function such as uthread_yield or an equivalent. There is no timer in user space that can preempt the thread at an arbitrary point. If the running thread never yields (because it is CPU-bound and does not know it should cooperate), no other thread in the process will ever run.

The consequence is that most user-level thread libraries require the programmer to insert yield points at appropriate places in the code. This is a form of cooperative multitasking at the thread level, and it is why early user-level thread systems were sometimes called "green threads" — they depended on the application's cooperation for any kind of parallelism.

4.5 The blocking problem

This is the fundamental limitation of the user-thread model. Suppose thread 1 calls read on a slow device. From the kernel's point of view, the calling process is executing a system call that cannot complete immediately, so the kernel blocks the process. The process moves from Running to Blocked in the state model from Lesson 3.2.

But now consider what this means for the other user threads of the same process. Thread 2 and thread 3 are runnable — from the library's point of view, they are sitting on the library's run queue, ready to execute. But the library is not running. It cannot schedule them, because the library itself is user-space code inside the blocked process. The kernel has suspended the entire process, and with it everything inside it, including the scheduler that might have switched to thread 2.

The consequence is stark: a blocking system call in any one user thread blocks the entire process. Not just the thread that made the call — the whole thing, including all the other threads. From the kernel's perspective this is correct behaviour, because the kernel only knows about the single process. But from the application's perspective it is a serious failure: a design that was supposed to give concurrent execution stops making progress as soon as any thread touches a slow device.

4.6 Workarounds for the blocking problem

Because the fundamental design has a serious limitation, researchers and library designers developed a set of techniques to work around it. All of them aim to prevent the process from ever making a blocking system call — because if the process never blocks, the library is always running and can always reschedule.

Non-blocking I/O

Replace every blocking operation with a non-blocking variant. On Linux, this means opening files and sockets with O_NONBLOCK, using select or epoll to discover which descriptors are ready, and never issuing a call that would block. The library, when it detects that an operation would block, marks the calling thread as blocked and switches to another. When the kernel later reports the descriptor as ready, the library can wake the waiting thread. This works, but it requires that every blocking operation in the program be replaced with a non-blocking equivalent — including inside libraries the program links against, which the programmer may not control.

Wrapper libraries (jacketing)

Interpose a wrapper around every blocking call. Before calling the underlying system call, the wrapper checks whether it is safe to proceed. If not, it registers the calling thread with the thread library as blocked and yields to another thread. When the operation completes, the library unblocks the original thread. This is called "jacketing" because the wrapper is a small jacket around the real call. It is transparent to the caller, but it is not always possible to interpose on every call — a statically-linked library that internally blocks may bypass the wrapper entirely.

Scheduler activations

A more sophisticated technique, described in the Draves paper. The kernel cooperates with the user-level thread library by notifying it when the underlying kernel thread is about to block. The library can then switch to another user thread before the kernel suspends the process. This gives the best of both worlds — user-level scheduling with kernel-mediated notifications — at the cost of a more complex kernel interface. Scheduler activations were implemented in research systems but never widely adopted in production.

Hybrid models (many-to-many)

Use multiple kernel threads, each running a set of user-level threads. The library multiplexes its user threads onto the available kernel threads. If one user thread blocks in the kernel, only the kernel thread it was running on is blocked; the others continue running their own user threads. This is the model used by Solaris's original threading library and by many research systems. It combines the efficiency of user-level switching with partial protection against blocking, at the cost of significant implementation complexity.

Each workaround has its own cost. Non-blocking I/O requires rewriting every operation. Jacketing requires interposing on system-call boundaries. Scheduler activations require kernel cooperation. Hybrid models require sophisticated libraries. None of them entirely solves the problem: even with all of them applied, there are cases where the process will still block and take all its user threads down with it.

4.7 The multicore limitation

A second fundamental limitation of pure user threads is that they cannot run on multiple cores simultaneously. Because the kernel sees only one schedulable entity per process, and because the library's switch is a user-space operation that runs on whichever core the process happens to be on, at most one user thread of a process can execute at any instant — even if the machine has many cores available.

This was not a serious concern when user threads were invented (in the 1980s, when single-core machines were the norm). It is a fatal flaw on modern hardware. A program that wants to exploit four cores cannot do so with pure user threads; it must either create separate processes or use kernel-level threads that the kernel can schedule across cores independently.

Lesson 4.4 examines the multicore implications in detail, and Lesson 4.3 explains how the kernel-thread model addresses them. For now, the important point is that the multicore era made the user-thread model's limitations more severe than they had been in the past.

4.8 Advantages of user threads

Despite the two fundamental limitations, user threads have real advantages. Some of these keep the model alive in specialised contexts; all of them motivated the design of hybrid systems that capture the best of both worlds.

4.9 Historical development and modern relevance

The user-thread model dominated academic threading research in the late 1980s and early 1990s. Its canonical implementations include:

• Mach C-threads. The user-level thread library for the Mach microkernel. Its design and its interactions with kernel threads were examined in the Draves paper and influenced nearly every subsequent library.
• Sun's green threads. The original Java virtual machine ran threads in user space to avoid the portability problems of kernel threads across platforms. It was replaced in Java 1.2 by a native-threads implementation, for the same reasons discussed in this lesson.
• GNU Pth. A portable user-level thread library for UNIX-like systems, still available today.
• ucontext-based libraries. Many small libraries and research prototypes have been built on the POSIX ucontext family over the years.

Modern systems rarely use pure user threads for general-purpose concurrency, because the blocking and multicore problems are too severe. But the model has not disappeared — it has been reincarnated in a different form. Modern language runtimes such as Go (goroutines), Erlang (processes), and Rust's async ecosystem implement their own lightweight tasks in user space and multiplex them onto a smaller number of kernel threads using exactly the techniques described in this lesson. The difference from the 1980s user-thread model is that the runtime controls the whole system — including every blocking call the program can make — so the workarounds are built in rather than bolted on.

4.10 When user threads are appropriate

Even with the blocking and multicore limitations, there are three contexts in which a pure or hybrid user-thread design remains appropriate.

1. When the workload is CPU-bound and single-threaded at the process level. If the tasks never block and the process will only ever run on one core, the multicore and blocking limitations do not apply. A user-thread library gives the fastest possible switches for the workload.
2. Inside a language runtime that controls all I/O. Go, Erlang, and Node.js are all examples. The runtime has replaced the blocking calls with its own non-blocking wrappers, so the process never blocks in a way the library cannot handle. In this context, user threads become a first-class scheduling abstraction.
3. For resource-constrained environments. When the per-thread kernel resources of the kernel-thread model (kernel stacks, task_structs) are too expensive, user threads provide a much lighter alternative. This is why some embedded systems and microcontrollers use user-thread libraries.

In all three contexts, the user-thread model is used with full awareness of its limitations, in a situation where those limitations do not matter or have been circumvented. Outside those contexts, the kernel-thread model is the correct default, and it is the subject of the next lesson.

05 Worked example

To make the user-thread model concrete, look at a small user-thread library and observe both its speed and its blocking behaviour.

#include <stdio.h>
#include <stdlib.h>
#include <ucontext.h>
#include <string.h>

#define MAX_THREADS 8
#define STACK_SIZE  65536

typedef enum { RUNNABLE, RUNNING, DONE } state_t;

typedef struct {
    ucontext_t  ctx;
    char        stack[STACK_SIZE];
    state_t     state;
    const char *name;
} uthread_t;

static uthread_t threads[MAX_THREADS];
static int       n_threads   = 0;
static int       current     = -1;
static ucontext_t main_ctx;   /* to return to when all threads finish */

/* Forward declaration */
static void uthread_switch(void);

/* Round-robin: pick the next RUNNABLE thread */
static void uthread_schedule(void) {
    for (int i = 1; i <= n_threads; i++) {
        int j = (current + i) % n_threads;
        if (threads[j].state == RUNNABLE) {
            int prev = current;
            current = j;
            threads[j].state = RUNNING;
            if (prev >= 0 && threads[prev].state == RUNNING)
                threads[prev].state = RUNNABLE;
            swapcontext(&threads[prev < 0 ? 0 : prev].ctx, &threads[j].ctx);
            return;
        }
    }
    /* No runnable thread: return to main */
    swapcontext(&threads[current].ctx, &main_ctx);
}

/* Yield: give up the CPU to the next runnable thread */
void uthread_yield(void) {
    uthread_switch();
}

static void uthread_switch(void) {
    int prev = current;
    threads[prev].state = RUNNABLE;
    uthread_schedule();
}

/* Thread entry wrapper */
static void uthread_exit(void) {
    threads[current].state = DONE;
    uthread_schedule();
}

/* Create a new thread */
int uthread_create(const char *name, void (*fn)(void)) {
    if (n_threads >= MAX_THREADS) return -1;
    int id = n_threads++;

    uthread_t *t = &threads[id];
    t->name  = name;
    t->state = RUNNABLE;

    getcontext(&t->ctx);
    t->ctx.uc_stack.ss_sp   = t->stack;
    t->ctx.uc_stack.ss_size = STACK_SIZE;
    t->ctx.uc_link         = &main_ctx;
    makecontext(&t->ctx, fn, 0);
    return id;
}

/* Run all threads to completion */
void uthread_run(void) {
    current = 0;
    threads[0].state = RUNNING;
    swapcontext(&main_ctx, &threads[0].ctx);
}

/* ---- Demo ---- */
static void worker_a(void) {
    for (int i = 0; i < 5; i++) {
        printf("A: %d\n", i);
        uthread_yield();
    }
    uthread_exit();
}

static void worker_b(void) {
    for (int i = 0; i < 5; i++) {
        printf("B: %d\n", i);
        uthread_yield();
    }
    uthread_exit();
}

static void worker_c(void) {
    for (int i = 0; i < 5; i++) {
        printf("C: %d\n", i);
        uthread_yield();
    }
    uthread_exit();
}

int main(void) {
    uthread_create("A", worker_a);
    uthread_create("B", worker_b);
    uthread_create("C", worker_c);
    uthread_run();
    printf("all threads complete\n");
    return 0;
}

$ gcc -Wall -Wextra -g -O2 -o uthreads uthreads.c
$ ./uthreads
A: 0
B: 0
C: 0
A: 1
B: 1
C: 1
A: 2
B: 2
C: 2
A: 3
B: 3
C: 3
A: 4
B: 4
C: 4
all threads complete

Total wall-clock for 3 million yields: 0.42 s
Switches per yield:                    ~2 (out and back)
Effective cost per switch:             ~70 ns

static void worker_a(void) {
    for (int i = 0; i < 5; i++) {
        printf("A: %d\n", i);
        if (i == 1) {
            printf("A: about to read from stdin (will block)\n");
            char buf[16];
            read(0, buf, sizeof(buf) - 1);
            printf("A: got input\n");
        }
        uthread_yield();
    }
    uthread_exit();
}

A: 0
B: 0
C: 0
A: 1
A: about to read from stdin (will block)

# ...the program hangs here until you press Enter,
# even though B and C have more iterations to run.

A: got input
B: 1
C: 1
A: 2
B: 2
C: 2
...
all threads complete

06 Interactive activity

07 Knowledge check

08 Unit discussion

09 Practice exercise

10 Summary

• A user-level thread is created, scheduled, and managed entirely by a library in the process's own address space. The kernel does not know the threads exist.
• The library maintains a thread control block (TCB) for each thread, holding the saved register context, a thread-private stack, a state field, and scheduling metadata.
• A user-thread switch is a pure user-space operation: save registers, restore registers, transfer control. No kernel trap, no mode switch. On Linux the mechanism is the POSIX ucontext family.
• Because the kernel does not schedule user threads, the library must. Most user-thread libraries are cooperative: the running thread must explicitly yield.
• The blocking problem is the model's central limitation: a blocking system call in any one user thread blocks the entire process, because the kernel suspends the process and the library cannot run.
• Workarounds for the blocking problem: non-blocking I/O, wrapper libraries (jacketing), scheduler activations, and hybrid many-to-many models. None is entirely satisfactory.
• The multicore limitation is the model's second central limitation: user threads of one process cannot run on multiple cores, because the kernel sees only one schedulable entity.
• User threads have real advantages — extremely fast switches, application-specific scheduling, portability, small per-thread footprint — which keep the model alive in specialised contexts and inside modern language runtimes.
• Go's goroutines, Erlang's processes, and Rust's async tasks are modern reincarnations of the user-thread idea. The model survived because the runtime controls the entire program, so the blocking and multicore problems can be solved by construction.

11 References

• Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 4, Section 4.3.
• Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapter 26. Available online at ostep.org.
• Draves, R., Bershad, B., Rashid, R., & Dean, R. Using Continuations to Implement Thread Management and Communication in Operating Systems. SOSP, 1991.
• Anderson, T. E., Bershad, B. N., Lazowska, E. D., & Levy, H. M. Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism. ACM TOCS, 1992.
• Linux man-pages: man 3 makecontext, man 3 swapcontext, man 3 getcontext, man 3 setcontext.
• GNU Pth project documentation, available at gnu.org/software/pth/.
• Butenhof, D. R. Programming with POSIX Threads, Chapter 1. Addison-Wesley, 1997.