Packages & signed repos

apt/dnf, GPG-signed repos, updates.

Every program on your Linux machine arrived somehow. On a well-run system, almost all of it came through a single front door called the package manager. It works like the receiving dock behind a busy restaurant. Ingredients don't appear in the kitchen by magic. They come from known suppliers, someone checks each delivery against the order, and only then does it go on the shelf. The package manager is that dock for software. It installs programs, and it keeps track of the other pieces each program needs to run (its dependencies), so you never end up with something half-installed and broken.

There are two big families of Linux, with two package managers that work the same way. Debian and Ubuntu use apt (the Advanced Package Tool), and the files it installs end in .deb. Red Hat, Fedora, and RHEL (Red Hat Enterprise Linux) use dnf (short for Dandified YUM, the modern package tool on that side), and its files end in .rpm (the Red Hat package format). The commands look different but line up almost one to one, so learning one teaches you the other. You'll see sudo in front of most of them, which means 'run this one command as the administrator,' something installing system software requires.

For a security engineer this is the interesting part. The package manager decides what code your machine will trust, and where that code is allowed to come from. Set it up carelessly and an attacker can feed you their software in place of the real thing. Set it up properly and a tampered or counterfeit package gets thrown out before it ever lands on disk. The rest of this lesson is about how that check works, and how people get it wrong.

Update and upgrade are two different jobs

New users mix these up constantly. apt update does not update your software. It refreshes the catalog. Your machine keeps a local list of every package each repository offers and which version is newest, and apt update downloads a fresh copy of that list. apt upgrade is the step that actually installs newer versions of the things you already have. The rhythm never changes: update to get the current catalog, then upgrade to place the order. A repository (repo for short) is a web server holding a large pile of packages plus an index that describes them.

That last line is worth reading every time. To see exactly what's behind, ask for the list.

Reading this before you upgrade tells you what you're about to pull in. A fix for something exposed, like openssh-server or an SSL library (the code that encrypts network connections), is a different level of urgency from a cosmetic bump. The word security in the origin (jammy-security) is your cue to act sooner rather than later.

Which package does this file belong to

Sometimes you're staring at a file and want to know where it came from. You find /usr/bin/ssh and you'd like to know whether it's part of a real, tracked package or something a person (or an intruder) dropped there by hand. The package manager keeps a record of every file it installed, so you can ask it in reverse: dpkg (the low-level Debian package tool) on one family, rpm (its Red Hat counterpart) on the other.

A file that no package claims is worth a closer look on a server you're investigating. The record goes further than names. On Red Hat systems, rpm -V compares every installed file against what the package originally shipped, like checking each item on the shelf against the original packing slip, and flags anything that has changed since.

Reading that line: the 5 means the file's contents no longer match what shipped, S means its size changed, T means its timestamp moved, and the c marks it as a config file. For sshd_config that's expected, because you edited it. For a plain binary like /usr/sbin/sshd, the same output would mean someone swapped the program out, which is exactly the kind of thing you want a tool to catch. On Debian and Ubuntu, dpkg --verify does the same job.

Signed repositories, and why the seal matters

Here's the question that makes or breaks all of this. When your machine downloads the nginx package, how does it know it got the genuine article from Ubuntu, and not a swapped-out copy from a hacked mirror, or from an attacker sitting on the network between you and the server? The answer is a digital signature. Think of the wax seal an old letter carried: one signet ring pressed it, anyone could recognize it, and a broken or wrong seal told you the letter had been opened. The digital version is built with GPG (GNU Privacy Guard, the standard tool for signing and encrypting on Linux).

A signature like this uses a matched pair of keys. The publisher keeps one of them secret forever (the private key) and hands the matching one to the whole world (the public key). They stamp their repository using the private key. Your machine checks that stamp using the public key. The math has a handy property: only the holder of the private key can produce a valid stamp, anyone with the public key can confirm it, and nobody can run the process backward to forge one. It behaves like a signature that's impossible to copy but easy for anyone to recognize.

One signature manages to cover an entire repository through a chain. The repo publishes a small index file (Ubuntu names it InRelease) and signs it. That signed index lists a hash for every larger index it points to. A hash is a short fingerprint calculated from a file's exact bytes, and changing a single byte changes the fingerprint completely. Those larger indexes in turn list a hash for every individual package. So when apt fetches nginx, it checks the package against the hash in the index, the index against the signature, and the signature against the publisher's key. Break any link in that chain and apt stops. One trusted key ends up vouching for everything beneath it.

When you add someone else's repository, you are adding their key to the set your machine trusts. The safer, modern way pins that key to one repository only, using a setting called signed-by.

deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable

The old approach, apt-key, threw every key you ever added into one global pile of trusted keys. A key you imported for a single vendor could then sign packages for anything on the system, including packages pretending to be the official Ubuntu ones. signed-by ends that by tying each key to a single repo, named right there in the sources file. It's the difference between handing a contractor the key to one room and handing them a master key to the whole building.

When the check fails, you see it. Import the wrong key, or none at all, and apt refuses to touch that repository.

Notice what happened. apt didn't shrug and install anyway. It disabled the repository. That refusal is the security boundary doing its job, and it's the exact behavior you never want to switch off to make an error go away.

Patching is the boring control that stops most breaches

Most successful attacks aren't clever. They reuse a known hole that a fix already exists for, against a machine nobody updated in time. These holes get logged in public with an identifier called a CVE (Common Vulnerabilities and Exposures), so defenders and attackers read the very same list. The gap between the day a patch ships and the day you install it is the window an attacker gets to work in. Keeping packages current closes that window, and per minute spent it is probably the highest-value security work you can do.

You don't have to do it by hand. Both families can apply security updates on their own: unattended-upgrades on Debian and Ubuntu, dnf-automatic on Fedora and RHEL.

Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}-security";
//      "${distro_id}:${distro_codename}-updates";
};
Unattended-Upgrade::Automatic-Reboot "false";

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

By default this pulls only from the security origin, so you get fixes without surprise feature changes, and it leaves the reboot decision to you. On an internet-facing box, an unpatched service is the single most common way it gets taken over, so leaving this switched on is usually the right call.

One habit to carry onto every server you inherit. Look at whose repositories it trusts, because that list is whose code can arrive as an update forever. An unfamiliar repo in there is a tidy way for a backdoor to keep itself supplied.

If you don't recognize a source in that directory, find out who added it and why before you run the next upgrade.

Try this

Work through “Patching is the boring control that stops most breaches” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: "curl | sudo bash" hands root to a stranger. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related