How Linux is put together
Kernel, userspace, distributions and the shell.
This lesson takes a Linux server apart into the pieces you will name every day: the kernel, the programs around it, the distribution that packages them, and the terminal and shell you type into. Each piece is shown with a command on the platform this course uses. By the end you can tell which part of the system produced a message, check what a server is running, and read the three error messages every beginner meets. That matters from the first day, because the first question in any problem is which layer is complaining.
The kernel and userspace
The kernel is the one program with full control of the hardware. It decides which program runs on the CPU next, gives each program its own memory, drives the disks and network cards, and checks permissions when a program opens a file. Everything else is userspace: the shell you type into, ls and cat, the SSH server, systemd, your own applications.
Userspace programs cannot touch the hardware directly. When they need something only the kernel can do, such as opening a file, they ask for it with a system call: a request with a fixed name like openat (open a file), read or write. You can watch those requests. strace runs a program and prints every system call it makes, and -e trace=openat limits the output to file opens. Here it watches cat print the file that holds the server's name.
Each line is one request and the kernel's answer after the =. The answer 3 is a file descriptor, the handle the kernel hands back for an open file; -1 followed by a code such as ENOENT (no such file or directory) is a refusal. Before cat reaches /etc/hostname it opens a cache of library locations, six shared libraries it was built against, two files under /proc and its message files (the lines left out). The last openat is the file you asked for, web01 is its content, and the final line reports that cat exited with status 0.
The library paths contain aarch64-linux-gnu because the lab server has an ARM processor. On an x86_64 server the same libraries live in /usr/lib/x86_64-linux-gnu, and the rest of the output is the same.
When it boots, the kernel starts one userspace program itself and gives it process ID (PID) 1. It also runs work of its own as kernel threads, which ps shows in square brackets.
PID 1 is systemd, run from /usr/lib/systemd/systemd; the options after the path record how it was started and can be ignored here. systemd starts every service on the machine, and it has a lesson of its own later in the course. PID 2, kthreadd, is the kernel thread that creates the kernel's other threads. Both show a parent process ID (PPID) of 0 because nothing in userspace started them. Every other process on the server has a parent you can follow back to one of these two.
Distributions, and the platform this course uses
The kernel alone is not something you can install and use. A distribution combines a kernel with userspace programs, a package manager to install and update them, default settings, and security updates for a stated number of years. Ubuntu, Debian, Red Hat Enterprise Linux (RHEL), Rocky Linux and Fedora are distributions. They share the kernel project and most of the programs, but they pick different versions, different defaults and sometimes different tools. Two commands tell you what you are on: /etc/os-release names the distribution, and uname -srm prints the kernel's name, its release and the processor architecture.
This course uses Ubuntu Server 26.04.1 LTS, the first update of the 26.04 long-term support release, and every terminal in it was produced by running the commands on that release. LTS releases are the ones most servers run, because they receive security updates for years. Many companies run RHEL instead. Where RHEL 10 differs in a way that changes a command, a file name or a default, the lessons say so and show output from Rocky Linux 10.2, a rebuild of RHEL 10.2 from the same source code.
Both are Linux, yet the kernel versions are far apart: Ubuntu 26.04 ships a 7.0 kernel, RHEL 10 a 6.12 kernel that Red Hat maintains with its own fixes. The differences you will meet most often are the package manager (apt on Ubuntu, dnf on RHEL), the security module (an extra layer of access rules in the kernel: AppArmor on Ubuntu, SELinux on RHEL), the names of some log files and services, and the two Ubuntu changes in the next section.
To follow along you need a practice server you can break and rebuild, never one that matters. Either of two kinds works. The first is a virtual machine on your own computer: install any virtualisation program, create a machine with 2 CPUs, 2 to 4 GiB of memory and a 20 GB disk, and install it from the Ubuntu Server 26.04 LTS image published on ubuntu.com. The second is the smallest Ubuntu Server 26.04 instance a cloud provider offers. A Rocky Linux 10 machine matches the RHEL notes.
New on Ubuntu 26.04: Rust coreutils and sudo-rs
The basic file and text commands (ls, cat, cp, mv, rm and about a hundred more) come from a collection called coreutils. For decades every major distribution used GNU coreutils. Ubuntu 26.04 is the first Ubuntu LTS to ship uutils coreutils instead, a reimplementation written in Rust, a language that rules out whole classes of memory bugs. You can see it in where the commands point.
/usr/bin/ls is a symbolic link (a pointer to another file, covered in the lesson on files, directories and links) into /usr/lib/cargo/bin/coreutils, where the Rust commands live. cp is different: Ubuntu points cp, mv and rm, the commands that walk and change whole directory trees, back to GNU for now, and df and true as well. Every GNU command is also installed under a gnu prefix, such as gnuls and gnucat. Nearly everything behaves the same. Error wording, help text and a few rarely used options can differ, so when a script written for GNU tools misbehaves on 26.04, running the gnu version of the command is the quickest test.
sudo, the command that runs a single command with administrator rights, changed the same way. Ubuntu 26.04 uses sudo-rs, a Rust rewrite that reads the same /etc/sudoers configuration, and keeps the original sudo installed as sudo.ws.
/usr/bin/sudo leads, through Debian's alternatives system (a set of links that lets several programs provide one command), to /usr/lib/cargo/bin/sudo. You type sudo exactly as before. RHEL 10 has neither change: its ls is the GNU program itself and its sudo is the original.
Terminal, SSH session and shell
Three different programs sit between your keyboard and the server, and error messages name them, so keep them apart. The terminal is the application on your own computer that draws text and sends your key presses. SSH is the encrypted connection that carries those key presses to the server and the output back. On the server, the SSH daemon (a daemon is a program that runs in the background, waiting for work) gives your session a pseudo-terminal, a kernel device with a name like /dev/pts/0 (the tty command prints it), and starts your login shell connected to it.
How you reach that shell depends on your practice machine. A virtual machine gives you a console window: log in with the user name and password you chose during the installation. A cloud instance is reached over SSH: the provider shows the command to use, usually ssh USER@ADDRESS, which you type into a terminal on your own computer (Windows, macOS and Linux all include ssh). How SSH works, and how to set up keys, is the subject of the lesson "Connecting with SSH" near the end of the course; until then, use the console or the command your provider gives you.
The shell is the program that reads each line you type, works out which program you mean and starts it. On Ubuntu and RHEL the shell you get when you log in is normally bash. It is an ordinary process: the special variable $$ holds its PID, and $0 holds the name it was started under. When a terminal in this course shows several lines after one $, type them one after another, pressing Enter after each; the output of all of them follows below.
The leading dash in -bash marks a login shell, the one started for you when you log in. Everything you type goes to this process first, and it starts a new child process for each program you run.
The shell prints a prompt when it is ready for the next command. On Ubuntu it looks like deploy@web01:~$: the user, the host, the current directory (~ means your home directory) and a $ that marks an ordinary user. RHEL shows the same facts as [deploy@rocky10 ~]$. The root account, the administrator that every permission check lets through, gets # instead of $.
The examples run as an ordinary user called deploy on a server named web01, so the prompts read deploy@web01; read them as your own user and host name. Like the first account an installer or a cloud image creates, deploy is allowed to use sudo. Every terminal shows $, and commands that need administrator rights start with sudo. If you ever see # in your own prompt, slow down: nothing will stop a mistake.
On a server installed from the Ubuntu Server image, sudo asks for your own password (not root's) and then remembers it for a few minutes; nothing appears on screen while you type it. Cloud images often set up their default account without that prompt. The deploy account that recorded the terminals in this course is configured so that sudo asks no password, so the labs could run unattended; that is the only reason you never see a prompt in them.
The first three errors, and $?
Every command finishes with an exit status, a number the shell keeps in $?. Zero means success and anything else means failure; the program chooses the number. Check it with echo $? straight after the command, because the next command replaces it.
No such file or directory means nothing exists at that path: usually a typo, a relative path typed from the wrong directory, or a file that lives on another machine. cat passed on the kernel's ENOENT and exited with 1.
Permission denied means the file exists but the kernel refused the request, and the strace line shows exactly that: openat returned EACCES, and cat printed the kernel's answer and exited with 1. /etc/shadow holds password hashes and only root and the shadow group may read it. The fix is never to loosen the file's permissions. Ask whether you should read the file at all, and if so, use sudo.
command not found comes from the shell itself. No program with that name exists in the directories the shell searches (the environment lesson shows how that search works), so nothing ran, and bash sets the status to 127. When commands come from a script, as they did for the lab that recorded these terminals, bash puts its own name and the line number in front of the message. Typed at an interactive prompt on Ubuntu, the same typo gets an answer from the command-not-found helper instead, which suggests the command or package you probably meant.
Try this
On your practice machine, as your ordinary user, run strace -e trace=openat ls /root. Find the line where the kernel refused, and the message ls printed from it. Then run ls /root followed by echo $?: ls reports 2 here, not 1, because it reserves 2 for serious trouble such as a directory named on the command line that it cannot open. Finish with cat /etc/os-release and uname -srm, and note the distribution, the kernel release and the architecture of the machine you will use for the rest of the course.
Takeaway
When a command fails, read the message for which layer is speaking (the shell, the program, or a kernel refusal the program passes on) and check echo $? before running anything else. When the message is not enough, strace shows the kernel's side of the conversation.