Shell and Command Line

01 Introduction

Before graphical desktops, every interaction with a computer was textual. A user typed a command; the machine responded. That interface — the shell — is not a relic of an earlier era. It remains the most powerful and efficient way to interact with a UNIX-like system, and it is the primary interface you will use for every laboratory and programming assignment in this course.

The reason the shell endures is composability. Graphical applications are islands: to get data from one into another, you copy and paste, or you rely on a plugin that someone else wrote. Shell commands are small, focused tools that read from standard input and write to standard output, and the shell's redirection and pipeline mechanisms let you connect them in arbitrary ways. A pipeline of three commands can accomplish what would take a custom program in any other environment — often without writing a single line of code.

This lesson gives you the practical skills you need: moving around the file system, running and finding commands, managing files and directories, and combining tools with redirection, pipes, and simple scripts. The lessons that follow will build on this foundation to cover users, permissions, and the development environment.

02 Learning objectives

This lesson contributes to U2.2 and to course learning outcome CLO6.

03 Required reading

• Stevens & Rago — Advanced Programming in the UNIX Environment, 3rd ed., Sections 1.4–1.6 (files and directories, input and output, programs and processes). The classical presentation of how the shell interfaces with the kernel.
• Shotts, W. — The Linux Command Line, 5th ed., Chapters 1–6 (navigation, file operations, redirection, and shell basics). Freely available online.
• Man pages: man 1 bash, man 1 ls, man 1 cp, man 1 grep. The manual is the definitive reference for every command discussed in this lesson.

04 Core content

4.1 What a shell is

A shell is a program that reads commands from the user, interprets them, and executes them. It is a user-space program — not part of the kernel — and it communicates with the kernel through the system-call interface. When you type ls, the shell:

1. Reads the line from standard input.
2. Parses it into a command name and arguments.
3. Searches the PATH for an executable named ls.
4. Calls fork to create a child process.
5. Calls execve in the child to replace its image with the ls program.
6. Calls wait to wait for the child to terminate.
7. Prints the next prompt.

This is the fork–exec–wait sequence from Lesson 2.1 in action. Every command you type — from ls to gcc to a script — runs through this mechanism.

4.2 Interactive vs. non-interactive shells

A shell can run in two modes:

• Interactive. A terminal is attached, the shell prints a prompt, and it reads commands from the user. This is what you get when you open a terminal or log in via SSH.
• Non-interactive. The shell reads commands from a file (a script) or from standard input and executes them without prompting. This is how shell scripts work — a text file containing commands that the shell executes in sequence.

The syntax is identical in both modes; the difference is only whether a prompt is displayed and a terminal is attached.

4.3 Command structure and the PATH

A shell command has a simple form:

command [option]... [argument]...

For example:

ls -l /home/alice
#   │  │  └─ argument (path)
#   │  └─── option (long listing)
#   └───── command (list directory)

Options come in two forms:

• Short options — single letter, prefixed with a single dash: -l, -a, -h. Multiple short options can be combined: ls -lah means ls -l -a -h.
• Long options — a word, prefixed with two dashes: --all, --human-readable. Long options are usually more readable and easier to remember.

When you type a command, the shell searches for an executable file with that name in a list of directories called the PATH. You can see the current PATH:

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# The directories are separated by colons.
# The shell searches them left to right and runs the first match.

The which command tells you which file the shell would run for a given command name:

$ which ls
/usr/bin/ls

$ which gcc
/usr/bin/gcc

$ which notacommand
(no output; the command was not found)

4.4 File-system navigation

The file system is a tree, and every file has a unique path. There are two kinds of path:

• Absolute — starts at the root, always begins with /. Example: /home/alice/notes.txt.
• Relative — interpreted from the current working directory. Example: notes.txt or ../bob/README.

Two special directory names appear everywhere:

• . — the current directory.
• .. — the parent directory.

Three commands are fundamental for navigation:

pwd — print working directory

Prints the absolute path of the current directory.

cd — change directory

Changes the current working directory. cd path moves to path; cd with no argument moves to your home directory; cd - returns to the previous directory.

ls — list directory contents

Lists the files in a directory. Common options: -l (long format), -a (include hidden files), -h (human-readable sizes), -t (sort by modification time), -R (recursive).

$ pwd
/home/alice

$ ls
docs  notes.txt

$ ls -la
total 24
drwxr-xr-x 4 alice alice 4096 Oct 15 10:22 .
drwxr-xr-x 3 root  root  4096 Oct 14 09:11 ..
-rw-r--r-- 1 alice alice  220 Oct 14 09:11 .bashrc
drwxr-xr-x 2 alice alice 4096 Oct 15 10:22 docs
-rw-r--r-- 1 alice alice   18 Oct 15 10:20 notes.txt

$ cd docs
$ pwd
/home/alice/docs

$ cd ..
$ pwd
/home/alice

4.5 File and directory management

The core file-management commands:

4.6 Getting help

There are several ways to find out what a command does and how to use it:

• Man pages. man ls shows the reference manual for ls. The man page is divided into sections; section 1 is user commands, section 2 is system calls, section 3 is library functions, and so on.
• --help. Many commands support a --help option that prints a short usage summary.
• info. GNU commands often provide a longer, hyperlinked manual accessible with the info command.
• apropos. Searches the short descriptions of all man pages for a keyword. Useful when you do not know the name of the command you need: apropos "copy file".

4.7 Redirection

Every process started by the shell has three standard streams:

Standard input (stdin, fd 0)

Where the process reads its input. By default, the terminal keyboard.

Standard output (stdout, fd 1)

Where the process writes normal output. By default, the terminal display.

Standard error (stderr, fd 2)

Where the process writes error messages. By default, also the terminal display — but a separate stream.

The shell can redirect these streams before starting a process:

• > file — redirect stdout to a file, overwriting its contents.
• >> file — redirect stdout to a file, appending to it.
• < file — redirect stdin from a file.
• 2> file — redirect stderr to a file.
• 2>&1 — redirect stderr to wherever stdout is currently going. Combined with >, this captures both streams in one file: cmd > out.txt 2>&1.
• > /dev/null — discard output by redirecting it to the special "null device" that throws away everything written to it.

# Save the listing of /etc to a file
$ ls /etc > etc-contents.txt

# Count the lines in that file
$ wc -l etc-contents.txt
142 etc-contents.txt

# Append another listing
$ ls /usr >> etc-contents.txt
$ wc -l etc-contents.txt
158 etc-contents.txt

# Save only the errors from a build
$ gcc broken.c 2> errors.txt

# Save both the output and the errors, interleaved
$ gcc -Wall broken.c > build.log 2>&1

4.8 Pipes

A pipe, written |, connects the stdout of one command to the stdin of the next. This is the mechanism that makes the shell truly powerful: rather than writing a program to do everything, you chain small tools that each do one thing well.

# Count how many files are in /etc
$ ls /etc | wc -l
142

# Find all lines in a log file that contain "error",
# then count how many there are
$ grep "error" /var/log/syslog | wc -l
37

# Sort the list of users on the system, remove duplicates, and page through it
$ cut -d: -f1 /etc/passwd | sort | uniq | less

# Show the ten largest files under /var in human-readable sizes
$ du -ah /var 2>/dev/null | sort -rh | head -10

Each command in a pipeline runs as a separate process, connected by a kernel-managed pipe (a buffer the kernel provides for interprocess communication). The commands run concurrently: as soon as the producer writes to the pipe, the consumer can begin reading, without waiting for the producer to finish. This streaming behaviour is what makes pipelines efficient even on large data.

4.9 Command substitution and variable expansion

The shell expands certain patterns before running a command:

• Variable expansion. $VAR is replaced by the value of the variable. $PATH, $HOME, $USER are common examples. Use ${VAR} when the variable name would otherwise be ambiguous.
• Command substitution. $(command) runs command and substitutes its output. For example, mkdir backup-$(date +%F) creates a directory named backup-2024-10-15.
• Arithmetic expansion. $(( expression )) evaluates an arithmetic expression: echo $(( 3 + 4 )) prints 7.
• Globbing. * matches any sequence of characters, ? matches any single character, and [abc] matches any character in the set. For example, rm *.tmp removes every file ending in .tmp. Note that globbing happens in the shell, not in the command — the shell expands *.tmp into the matching filenames before running rm.

4.10 Shell scripts

A shell script is a text file containing a sequence of shell commands. To run it:

1. Write the commands to a file.
2. Make the file executable: chmod +x script.sh.
3. Run it: ./script.sh or bash script.sh.

A minimal script:

#!/bin/bash
# The first line is the "shebang": it tells the kernel
# which program should interpret this script.

echo "Hello from a shell script"
echo "Today is $(date +%F)"
echo "The current user is $USER"

The #!/bin/bash line is essential. When you run ./script.sh, the kernel reads the first two bytes; if they are #!, it treats the rest of the line as the path to the interpreter. The kernel then runs /bin/bash with the script path as its argument.

Scripts can use control flow, variables, and conditionals:

#!/bin/bash

# Arguments
NAME="$1"
if [ -z "$NAME" ]; then
    echo "Usage: $0 <name>" >&2
    exit 1
fi

# Loops
for FILE in *.txt; do
    if [ -f "$FILE" ]; then
        LINES=$(wc -l < "$FILE")
        echo "$FILE: $LINES lines"
    fi
done

This script uses:

• Positional parameters. $1 is the first argument, $0 is the script name, $@ is all arguments.
• Test expressions. [ -z "$NAME" ] tests whether NAME is empty; [ -f "$FILE" ] tests whether FILE exists and is a regular file.
• Exit codes. Every command returns an exit code; 0 means success, non-zero means failure. exit 1 ends the script with code 1.
• Loops and conditionals. for, if, while are the standard control-flow constructs.

Shell scripts are covered in far more depth in the next lesson and in Lab 1. For now, the goal is to recognise the syntax and be able to read a simple script when you encounter one.

4.11 Scripts in this course

You will write shell scripts for at least three purposes in this course:

• Lab automation. Each laboratory has a set of commands to run and outputs to collect; wrapping them in a script makes the workflow reproducible.
• Test harnesses. When you implement a scheduler, a memory allocator, or a file-system simulator, you will want a script that runs a set of test cases and reports the results.
• Experiment automation. When you measure context-switch cost or page-fault rates, a script that runs the experiment across a range of parameters and collects the results in a consistent format saves an enormous amount of manual work.

The best way to learn is to type commands by hand. Every command you run at the prompt is a line you could put in a script later.

05 Worked example

To see the power of composition in action, let's build a small but realistic pipeline from scratch.

tr -cs '[:alpha:]' '\n' < input.txt \
  | tr '[:upper:]' '[:lower:]' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -10

    892 the
    521 and
    489 of
    418 to
    389 a
    312 in
    287 is
    265 that
    231 it
    208 for

#!/bin/bash
# topwords — print the N most common words in a file
# Usage: topwords FILE [N]

FILE="$1"
N="${2:-10}"   # default to 10 if not given

if [ ! -f "$FILE" ]; then
    echo "topwords: no such file: $FILE" >&2
    exit 1
fi

tr -cs '[:alpha:]' '\n' < "$FILE" \
  | tr '[:upper:]' '[:lower:]' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -n "$N"

06 Interactive activity

07 Knowledge check

grep "warning" data.txt | wc -l

grep -c "warning" data.txt

08 Unit discussion

09 Practice exercise

10 Summary

• A shell is a user-space program that reads commands, interprets them, and executes them by forking child processes and calling execve.
• The file system is a tree; paths are either absolute (starting from /) or relative (starting from the current working directory). . and .. refer to the current and parent directories.
• Commands are found by searching the directories listed in the PATH environment variable.
• The core file-management commands are ls, cd, pwd, touch, mkdir, cp, mv, and rm.
• Standard input, standard output, and standard error are three separate streams. Redirection (>, >>, <, 2>, 2>&1) lets you control where each goes.
• Pipes (|) connect the output of one command to the input of the next, allowing small tools to be composed into larger data-processing workflows.
• The shell expands variables ($VAR), command substitutions ($(cmd)), arithmetic ($(( ... ))), and globs (*, ?, [...]) before running a command.
• A shell script is a text file containing commands; the #!/bin/bash line identifies the interpreter, and the script becomes executable with chmod +x.

11 References

• Stevens, W. R., & Rago, S. A. Advanced Programming in the UNIX Environment, 3rd ed., Sections 1.4–1.6. Addison-Wesley, 2013.
• Shotts, W. The Linux Command Line, 5th ed., Chapters 1–6. No Starch Press, 2019. Freely available at linuxcommand.org.
• Newham, C., & Rosenblatt, B. Learning the bash Shell, 3rd ed. O'Reilly, 2005.
• Robbins, A., & Beebe, N. H. F. Classic Shell Scripting. O'Reilly, 2005.
• Linux man pages: man 1 bash, man 1 ls, man 1 cp, man 1 grep, man 1 find, man 7 glob.