Patching and update strategy

Automatic updates, restarts and reboots.

This course takes a default Ubuntu Server 26.04 installation and hardens it one control at a time. For each control you see the threat it answers, the configuration, a command that proves it is active, what it costs in operation, and how to undo it, plus whose advice it is: the upstream default, Ubuntu's or Red Hat's recommendation, a CIS Benchmark item, or SecOpsLog's own judgement. Where RHEL 10 differs, a transcript from Rocky Linux 10.2 (a rebuild of RHEL 10.2) shows how. The course assumes Linux essentials.

Patching comes first because every later control assumes the software underneath carries its published security fixes, and a flaw with a public advisory takes little skill to exploit. By the end of this lesson you can check what a fresh install already patches, apply and verify an update, make sure running services and the kernel actually use the fixed code, choose a reboot policy, and roll a bad update back.

Before you start

Practise on a virtual machine you can snapshot and throw away, not on a server anyone depends on. Several lessons change how people log in (sudo rules, PAM, the SSH server, the firewall), and one wrong line there can lock every account out. Before the accounts lesson, find out how you would reach the machine without SSH: your provider's web or serial console, or the hypervisor's.

The lab's administrator, deploy, has password-less sudo so the labs run unattended. On your machine sudo asks for your password, and once the PAM lesson turns on lockout, mistyped sudo passwords count toward it. Each lesson creates its own hard-* users, units and files, and shows the commands or says exactly what was set up. Terminals titled Rocky Linux need a second, RHEL-family VM (Rocky Linux or AlmaLinux 10); without one, read those parts.

For every change that can cut off logins, use one routine. Keep a second session open with a root shell (sudo -i). Before you apply the change, arm an automatic undo, for example sudo systemd-run --on-active=5min --unit=undo-change sh -c 'COMMANDS THAT UNDO IT'; the nftables lesson runs this pattern. Test a new login from a new terminal, then cancel the undo (sudo systemctl stop undo-change.timer) and close the spare session.

CIS, the Center for Internet Security, publishes configuration benchmarks per operating system. Level 1 items are meant to be safe on almost any server; Level 2 items add defence in depth and are more likely to break something. The course reads CIS items from scap-security-guide (SSG) 0.1.82 on Rocky Linux, the ComplianceAsCode project's machine-readable content, whose CIS profiles encode the CIS Red Hat Enterprise Linux 10 Benchmark v1.0.1. On more than one server, treat every drop-in in this course as a file for your configuration management: run the tool's own check before installing it (sshd -t, visudo -cf, nft -c -f, rsyslogd -N1), create what a file depends on before the file (a group before AllowGroups), and roll out to a canary first.

What a fresh install already does

Ubuntu Server ships unattended-upgrades installed and enabled: a job that installs security updates with nobody logged in. Two systemd timers drive it. This is Ubuntu's default, shown here on an untouched install.

apt-daily.timer refreshes the package lists, like apt update, and apt-daily-upgrade.timer runs unattended-upgrade, which installs what it is allowed to. The two "1" values mean "every day". The upgrade timer fires at 06:00 plus a random delay of up to an hour, so a fleet does not hit the mirrors in the same minute; that is why the NEXT times are not round. What counts as allowed is set in 50unattended-upgrades.

Each entry is an origin and a pocket (a section of the archive). -security holds security fixes. The release pocket is listed because a security fix sometimes needs a new dependency from it. -updates, the pocket for bug fixes, is commented out: a default server gets security fixes automatically and ordinary bug-fix updates only when someone runs apt upgrade. SecOpsLog recommends keeping that split. Security updates are small, targeted backports; bug-fix updates are the ones worth testing on a few machines first. One case stops the nightly job: when a new package version changes a configuration file you edited (a conffile), dpkg needs a person to choose, so unattended-upgrade skips the package and logs "Package ... has conffile prompt and needs to be upgraded manually". The security fix in it waits until someone reads that log, so alert on the line, and prefer drop-in directories over editing packaged files, as this course does.

Ubuntu also phases bug-fix updates (Linux essentials showed how); security updates are never phased. What matters on a fleet is that phasing makes identical servers differ:

This machine is outside the first 75% for the new rust-coreutils, so apt defers it. The share a machine falls into is computed from /etc/machine-id, the source package name and the version, so two identically built servers can run different versions for a few days. If a fleet must stay identical, apt_preferences(5) documents APT::Get::Always-Include-Phased-Updates and Never-Include-Phased-Updates; you then stage the rollout yourself.

Applying an update, and restarting what used it

A process keeps running the library it loaded until it restarts (Linux essentials showed the deleted file in memory). needrestart, installed on Ubuntu Server by default, finds such processes after every apt run and, since Ubuntu 24.04, restarts the affected services itself. To have a real update to apply, put back the release-day version of libexpat1, an XML parser library that several system daemons load; this is also how to recreate the lab's starting point:

A security update for it is now waiting.

Seven packages are upgradable, but the dry run says the nightly job would install only libexpat1, the one that is also in -security. The rest come from -updates and wait for you. --dry-run is also how you check any change to the unattended-upgrades configuration before trusting it. Now run the job the way the timer does:

needrestart restarted polkit.service on its own. Three services were deferred. /etc/needrestart/needrestart.conf keeps a list of services it never restarts automatically: dbus, the system message bus, is on it and gets a helper script (restart.d/dbus.service) that also restarts the services depending on it, and networkd-dispatcher matches the qr(^network) entry that exempts networking daemons. unattended-upgrades is the program doing the upgrade. Those keep the old library until a manual restart or the next reboot, and needrestart in list mode is how you check.

sudo needrestart -r l lists what still runs old code and restarts nothing. The log under /var/log/unattended-upgrades/ and /var/log/apt/history.log answer "what changed last night".

The cost is that services restart unwatched, around 06:00. For a service that must only restart in a maintenance window, Ubuntu's announcement of the change, made for the 24.04 release, documents a drop-in such as /etc/needrestart/conf.d/myapp.conf containing $nrconf{override_rc}{qr(^myapp\.service$)} = 0; (the configuration is Perl, and qr() is a regular expression). Removing that file undoes it. To return to prompting instead of automatic restarts for everything, the same announcement says to remove -m u, needrestart's "Ubuntu mode" flag (needrestart(1)), from the hook in /etc/apt/apt.conf.d/99needrestart.

Kernels and the reboot gap

Some fixes cannot be picked up by restarting a service. A new kernel only runs after a reboot, and restarting libc or dbus users means restarting nearly everything. On Ubuntu, packages such as the kernel, libc6 and dbus create /run/reboot-required when they need one, and needrestart -b reports the kernel state in a form monitoring can parse.

KSTA: 1 means the running kernel is the newest installed; 2 or 3 would mean a newer one is waiting (README.batch.md in the needrestart package). ("ABI upgrades are not detected" in its output means it compares version strings only.) The previous kernel stays installed as vmlinuz.old, your way back if a new kernel does not boot. It is chosen at the console under "Advanced options" in the GRUB menu, which a default server hides:

With a hidden menu and no timeout, press Esc or hold Shift at the console while GRUB starts to bring the menu up (the GNU GRUB manual, GRUB_TIMEOUT_STYLE). That needs a console, so find yours before the day you need it.

The Rocky Linux lab machine shows the gap. A reboot clears it (dnf needs-restarting -r compares install times with the boot time), so the lab's setup reinstalls the newer kernel-core, already the boot default, as a fresh kernel update would leave it.

dnf needs-restarting -r exits 1 and names the package behind the verdict, kernel-core; after a real update it also lists libraries such as glibc. The machine runs 211.16.1 while 211.60.1 is installed and set as the next boot. Until someone reboots, the fixes in the new kernel protect nothing. RHEL keeps three kernels installed (installonly_limit=3 in /etc/dnf/dnf.conf), so the old one stays available.

You need a reboot policy, and there are three reasonable ones. Unattended reboots suit stateless servers behind a load balancer whose health checks absorb one missing node; Automatic-Reboot neither drains the node nor checks its peers. Clustered and stateful roles (databases, anything with a quorum) and single servers get a monitored window, one member at a time with a health check in between. Live patching shortens the gap for kernels. Canonical Livepatch, an Ubuntu Pro service, applies fixes for high and critical kernel vulnerabilities to the running kernel where it can; Canonical's documentation says it does not replace installing kernel updates and rebooting. Red Hat's kpatch does the same for a defined set of fixes with a standard subscription (longer with EUS); on RHEL 10 it starts with the 10.2 kernel, 6.12.0-211.16.1.

The default install lists Livepatch as available but is not attached to Ubuntu Pro, so nothing is live-patched. Ubuntu's default is also no automatic reboot. To choose one, add a drop-in that sorts after 50unattended-upgrades: apt reads /etc/apt/apt.conf.d in alphanumeric order, and a later setting replaces an earlier one.

// SecOpsLog: when an update leaves /run/reboot-required behind (a new
// kernel, libc6, dbus), reboot at 03:30 instead of waiting for a person.
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:30";

apt-config dump prints the merged configuration, so it proves the file was read. The ls is what your monitoring runs: no file, no reboot pending. The impact is a reboot's worth of downtime at 03:30 whenever such an update lands, with users logged in or not (Automatic-Reboot-WithUsers defaults to true). SecOpsLog advice: stagger the time per group of servers so they never all reboot together, and alert on how long /run/reboot-required has existed, not only on whether it does. To roll back, delete the file and check that apt-config dump no longer shows the setting.

On RHEL 10: dnf-automatic

RHEL installs nothing that patches on its own.

Red Hat's documentation installs dnf-automatic and enables one of its timers. The defaults download all updates and install nothing. dnf-automatic-install.timer installs regardless of apply_updates, and upgrade_type = security limits it to updates marked as security advisories, which mirrors Ubuntu's split. That limit is SecOpsLog's choice, not Red Hat's default.

The timer verifies itself in list-timers. Its reboot = never default leaves the kernel gap to you, just as on Ubuntu; when-needed is the unattended option. Rolling back is two steps: disable the timer, then undo the transaction that installed the package. Find that transaction by the package name, which lists only the transactions that touched it, newest first, and undo it by its ID.

Undo by ID, not with dnf history undo last: last is whatever transaction ran most recently on the host, which on a shared server may be someone else's install or removal. The < in the last column is dnf(8)'s mark that the RPM database was changed outside DNF before this transaction; it does not affect the undo. dnf history undo reverses any transaction, including an update that broke something, as long as the old packages can still be downloaded. Your edited automatic.conf survives as .rpmsave.

Rolling out, and rolling back

Automatic security updates on every server are the baseline. SecOpsLog advice for larger changes, such as bug-fix updates, new kernels and their reboots: let them reach machines in stages, so a bad update hurts a few of them first.

When an update does break something, find out exactly what changed, then go back one version and hold it so the nightly job does not reinstall it.

history.log names the command, the user and the old and new versions. --allow-downgrades lets apt downgrade without asking; apt-get(8) calls it dangerous outside deliberate cases like this one. It worked because 2.7.4-1 is the release version, which the archive keeps for the life of the release. The -updates and -security pockets list only their newest version, so an intermediate version comes back only from /var/cache/apt/archives or from Ubuntu's snapshot service (apt-get(8) --snapshot, for sources with Snapshot: enable); a fleet that needs reliable rollback mirrors its repositories. The hold works, and that is its danger: the dry run finds nothing to install, so the security fix you rolled back stays out until someone removes the hold. Record every hold with an owner and a date, and release it as soon as a fixed version ships.

Try this

On your Ubuntu lab machine, roll libexpat1 back as above, run sudo apt-mark hold libexpat1 and confirm that sudo unattended-upgrade --dry-run --verbose reports "No packages found that can be upgraded unattended". Release the hold and confirm that the package is listed under "Packages that will be upgraded". Apply it with sudo unattended-upgrade --verbose, then use sudo needrestart -r l to list which services still run the old library and restart them yourself, except dbus, which waits for a reboot. On Rocky Linux, run sudo dnf needs-restarting -r and explain its exit status.

Takeaway

An update counts only when the running process and the running kernel use it. After every patch run, check needrestart -r l or dnf needs-restarting -r, and decide your reboot policy before an urgent kernel fix forces the decision.

Related