The stack at a glance

Think of seven layers. Each passes work to the one below and results back up:

1. Terminal UI — window, font, scrollback (iTerm2, GNOME Terminal, Windows Terminal, VS Code integrated terminal).
2. PTY pair — pseudo-terminal master/slave; makes the shell believe it talks to a “serial device.”
3. Shell — bash, zsh, fish; reads a line, parses, dispatches commands.
4. ls process — userspace program from GNU coreutils (or BusyBox on minimal images).
5. C library (glibc/musl) — wraps syscalls: opendir, readdir, write, etc.
6. Linux kernel — syscall interface, VFS, scheduler, memory, permissions.
7. Storage — ext4, xfs, btrfs, NFS, overlayfs in containers—whatever backs the inode you opened.

  [Keys] → Terminal emulator → PTY master
                ↓ read()
  [Shell: bash]  parse "ls" → fork() → execve("/usr/bin/ls")
                ↓ syscalls
  [Kernel VFS]  → filesystem driver → disk / network FS
                ↓ write(1, ...)
  [PTY] → Terminal → pixels on screen

Everything below assumes a normal Linux session (local or SSH). macOS and WSL follow the same ideas with different paths and sometimes different utilities (ls from BSD vs GNU).

1. You type — what the terminal actually does

When you press l, the terminal emulator receives key events from the OS (X11, Wayland, macOS window server, or a remote SSH client decoding network packets). It does not send the letter directly to ls. It writes bytes into the master side of a pseudo-terminal (PTY).

A PTY is a kernel object that looks like a serial port pair:

• Slave (/dev/pts/3) — attached to your shell’s stdin/stdout/stderr.
• Master — held by the terminal emulator; it forwards your keystrokes to the slave and reads output to paint the screen.

The kernel’s line discipline (often “cooked” mode) sits on the slave: it handles echo (so you see characters), backspace, and optionally line editing until you press Enter. That sends a newline character (\n, sometimes \r\n depending on settings). Only then does the shell’s read() (or libreadline after you pressed Enter) return with a full line: ls\n.

If you use raw mode (vim, htop), the line discipline stops cooking input—each key can reach the program immediately. The shell uses cooked mode for normal prompts.

2. The shell wakes up: one line of input

Your shell process (e.g. bash, PID 4500) was blocked in read() on file descriptor 0 (stdin). The returned buffer might be exactly three bytes: l, s, \n—plus history and editing handled by readline before the line is submitted.

Next steps inside bash (simplified but accurate):

1. History — append to ~/.bash_history (unless HISTCONTROL ignores it).
2. Alias expansion — if alias ls='ls --color=auto', the line becomes that before parsing.
3. Tokenization — split into words respecting quotes: ls -la /tmp → words ls, -la, /tmp.
4. Parsing — build a simple command (no |, &&, redirects yet in our example).
5. Expansion — globs (*), variables ($HOME), command substitution—none apply to plain ls.

Bash checks whether ls is a shell builtin. On many systems ls is not builtin (unlike cd or echo on some shells). Run type ls—you will usually see ls is hashed (/usr/bin/ls) or similar.

3. Finding the program: PATH and execve

The first word is the command name. For an external command the shell searches directories in PATH (colon-separated list in the environment):

echo $PATH
# often: /usr/local/bin:/usr/bin:/bin

For each directory it tests /usr/local/bin/ls, then /usr/bin/ls, etc., until an executable file exists and passes permission checks (execute bit + directory traverse for others). The result is often cached in a hash table so the next ls skips the search.

Then the shell does the Unix dance:

• fork() — clone the shell process. Parent and child get different PIDs; child inherits open file descriptors (including stdin/stdout/stderr attached to the PTY).
• execve(path, argv, envp) — in the child only: replace the program image with /usr/bin/ls. Same PID, new memory layout, new code. If exec fails (file not found), the child exits with an error and the parent prints “command not found.”

Arguments passed might be:

argv[0] = "ls"          # convention: program name
argv[1] = NULL
envp    = environ       # PATH, HOME, LANG, LS_COLORS, ...

The parent shell typically blocks in wait4() until the child exits, then stores the exit status in $?.

4. Loading /usr/bin/ls: from bytes on disk to main()

ls is an ELF binary. The kernel’s loader (with help from the dynamic linker if the binary is dynamically linked):

1. Maps the executable and interpreter (/lib64/ld-linux-x86-64.so.2) into memory.
2. Loads shared libraries (libc.so.6, maybe libselinux, etc.) listed in .dynamic.
3. Relocations and symbol resolution run; constructors in libc fire.
4. Control jumps to _start → libc startup → main() in coreutils’ ls.c.

Static binaries skip the dynamic linker but are rare for distro ls. In containers you still hit the same exec path; the binary may be from BusyBox (/bin/ls applet) with a smaller codebase but similar syscalls.

5. Inside ls: logic you trigger with flags

GNU ls parses options (getopt_long): -l long format, -a almost all files, -h human sizes, --color when stdout is a TTY. It determines which directories to list—default “.” (current working directory) from the process cwd inherited from the shell.

For each directory it must obtain names and metadata:

• Name only — read directory entries (see syscalls below).
• ls -l — for each name, lstat() or stat() for mode, owner, size, mtime; may read passwd/group for names.
• Sorting — often in userspace after collecting entries; locale (LC_COLLATE) affects order.
• Colors — map file type and extension to ANSI escapes from LS_COLORS; enabled when isatty(1) is true (interactive terminal).

Finally it formats columns (or one name per line) into a buffer and writes to file descriptor 1 (stdout). Errors go to fd 2 (stderr)—e.g. “Permission denied” on a subdirectory with ls dir.

6. System calls: the boundary to the kernel

Userspace cannot read the disk directly. libc functions translate to system calls—the controlled API into the kernel (on x86-64 via the syscall instruction with a syscall number and arguments in registers).

Typical syscalls for plain ls in the current directory:

Older code paths use opendir/readdir in libc, which still bottom out in openat + getdents64 on Linux. Watching syscalls is the best way to see reality:

strace -e trace=openat,getdents64,write,close ls
# or full firehose:
strace -f ls 2>&1 | head -40

You will see the kernel return directory entry structures; libc may filter . and .. unless -a is set.

7. Kernel: VFS, inodes, and permissions

When ls asks to open ., the syscall enters the Virtual File System (VFS) layer. VFS does not care whether the backing store is ext4 on NVMe, an NFS mount, or overlayfs in a container—it provides uniform struct file operations.

1. Path lookup — walk components from the process root (or cwd): resolve . to an inode via the mount table.
2. Permission check — POSIX permissions (owner/group/other) and optionally MAC (SELinux, AppArmor). Fail → EACCES to userspace.
3. Directory read — filesystem-specific code reads the directory’s data blocks (or equivalent) and fills syscall buffers with (name, type, inode number) tuples.
4. Stat — for -l, load the inode: mode, nlink, uid, gid, size, timestamps.

The scheduler may run other processes between syscalls; your ls is not “in the kernel” continuously—it bounces userspace ↔ kernel per call.

8. Output path: from write() to pixels

write(1, ...) goes to stdout. Because the shell forked with the PTY slave still open as fd 1, bytes go to the line discipline and then to the PTY master. The terminal emulator read()s them, parses ANSI color sequences if present, updates the screen grid and scrollback, and composes a frame for the display server.

No network is involved for a local terminal. Over SSH, the same byte stream is encrypted inside sshd on the server and decrypted by your local terminal—the PTY is on the remote host; the visual stack on your laptop is still “terminal app → screen.”

9. Child exits; shell returns

ls calls exit(0). The kernel sends SIGCHLD to the parent shell. The parent’s wait4() returns with status 0; bash sets $?=0, may update job control state, and loops: print prompt (PS1), flush, block on read() again.

If you typed ls /nonexistent, ls might still exit 2 after printing an error on stderr—the shell still waits and records that code.

Variants that change the story

Timing: why it feels instant

A local ls on a small directory costs milliseconds:

• fork + exec — tens of microseconds to low milliseconds (mitigated by kernel optimizations and hash caching).
• Directory read — often cached in the page cache if you listed recently; cold reads hit disk.
• Terminal render — proportional to line count and font complexity.

Slow ls on huge directories (thousands of files) is usually userspace sorting and stat storms for -l, or network filesystem latency—not the shell typing path.

Observe it yourself (lab)

# Who is running?
ps -o pid,ppid,cmd -p $$,$(pgrep -n ls)

# Trace syscalls (Linux)
strace -f -o /tmp/ls.trace ls
less /tmp/ls.trace

# See ELF and libraries
file $(which ls)
ldd $(which ls)

# Compare builtin vs external
type cd ls
/usr/bin/time -f '%e sec' ls /usr/lib > /dev/null

On macOS, use dtruss or fs_usage instead of strace; the conceptual stack is the same, but ls is often the BSD implementation.

How this connects to the rest of your stack

Every CLI tool—kubectl, docker, terraform—follows the same fork/exec pattern unless it is a shell builtin. Containers add namespaces so the ls you run in a pod still sees its own rootfs and cgroup limits, but the syscall interface is the same Linux kernel on the node. Kubernetes debugging on a node often means crictl exec → shell → ls in a mount namespace.

For a broader Linux handbook (commands, permissions, systemd), see Linux in depth. For how containers reuse the kernel without a full VM, see Docker’s hidden side.

Further reading

• Michael Kerrisk — The Linux Programming Interface (processes, syscalls, filesystems)
• William Shotts — The Linux Command Line (shell and navigation)
• GNU coreutils manual — info ls and source at savannah.gnu.org
• Linux man pages — pty(7), fork(2), execve(2), getdents64(2), path_resolution(7)

Blog index · Linux in depth · Docker hidden side · Git & GitHub in depth

Back to blog list