The boot process & systemd-analyze

From firmware to a running system, timed.

A cold boot is a relay race. The firmware runs the first leg and hands a baton to the bootloader, the bootloader hands it to the kernel, the kernel hands it to systemd, and systemd sprints the rest of the way to a login prompt. Each runner only knows how to reach the next one. So when a machine won't come up, it didn't fail everywhere at once. It fumbled the baton at exactly one handoff, and knowing the legs of the race tells you which runner to go question.

That same chain is where the quietest persistence lives. Everything here runs before your monitoring wakes up, so an attacker who can plant code early gets it for free on every reboot. Learning to time and read the boot is how you fix slow starts and notice the thing that shouldn't be starting at all.

The Four Handoffs

Here are the legs, in order. Each has one job: get far enough to start the next one.

The firmware (the low-level code baked into the motherboard, either the older BIOS, Basic Input/Output System, or the modern UEFI, Unified Extensible Firmware Interface) runs first. It wakes the hardware, runs a quick self-check called POST (power-on self-test), and looks for something bootable. On a UEFI machine that means reading a small FAT-formatted partition called the EFI System Partition and running a bootloader program stored there. If Secure Boot is on, the firmware checks that bootloader's cryptographic signature before trusting it. That signature check is your first line of defense against a tampered boot chain.

Next the bootloader, almost always GRUB (GRand Unified Bootloader), takes the baton. GRUB is the menu you sometimes see listing kernels. Its real work is loading two files into memory: the kernel image (vmlinuz) and the initramfs, then handing the kernel a line of text called the kernel command line. That command line, assembled from /etc/default/grub into /boot/grub/grub.cfg, is where boot options live. Remember it. It is the single most useful lever you have when a box won't finish booting.

The kernel (the core of the operating system, the part that talks directly to the hardware) unpacks itself and brings up CPUs, memory, and devices. But it has a chicken-and-egg problem: the drivers it needs to read your real disk might live on that disk. The initramfs (initial RAM filesystem) solves it. It is a tiny throwaway root filesystem loaded straight into memory, carrying enough drivers to find the real root, including the tricky cases: encrypted volumes, LVM (Logical Volume Manager, which pools disks into flexible volumes), software RAID (Redundant Array of Independent Disks), and network storage. Once the initramfs mounts the real root, it switches over to it (switch_root) and starts the first real program.

That first program is PID 1 (process ID 1, the very first userspace process and the ancestor of every other one). On a modern Linux that is systemd (the system and service manager). systemd reads its goal, the default target (a named end-state, systemd's version of the old runlevels), usually graphical.target or multi-user.target. Then it walks the dependency graph and starts everything that target needs, running independent units at the same time instead of one after another. When the target is reached, you get a login prompt.

Putting A Stopwatch On Boot

systemd times every unit it starts, and systemd-analyze reads those numbers back to you. Start with the wide view, which splits total boot time across the stages you just met.

Firmware and loader you mostly can't change. The kernel phase is drivers and hardware init. Userspace, that last and usually largest chunk, is systemd starting your services, and it is the part you can actually shrink. To see which units ate the time, rank them.

Here is the trap. blame is a list of the slowest dishes by cook time, but dinner is served when the last dish on the critical path finishes, and several dishes cooked at the same time. A unit can be slow and still not delay boot at all, because everything waiting on it was already waiting on something else. So shortening the top of blame often changes total boot time by nothing. The honest number comes from the critical chain: the real path of this-waited-for-that that gated the finish.

Read the @ as the clock time when that unit kicked off, and the + as how long it then took to finish coming up. This chain, not the blame list, is what to optimize. The unit holding the finish line here is NetworkManager-wait-online, which started at fifteen seconds and then sat for six more waiting for the network to call itself up. cloud-init, the loudest name in blame, isn't on this path at all, so trimming it wouldn't move the total. Disabling that wait-online unit, or telling it which interface actually matters, is the real six-second win.

Reading What The Boot Actually Did

When boot goes wrong, the logs are your recording of the race. The systemd journal keeps them separately for each boot, so you can walk back through past starts.

journalctl -b (or -b 0) is this boot; journalctl -b -1 is the one before it. That previous-boot view answers a specific question: did the box reboot cleanly, or did something knock it over? A clean shutdown ends with orderly stop messages. A crash or power loss stops mid-sentence, and the next boot starts fresh. The other quick check is which units never came up.

systemctl --failed is the morning-after list: every unit that tried to start and gave up. On a healthy machine it should be empty. Anything on it is either your problem to fix or, once in a while, a service you didn't install and didn't expect. Hold that second thought. It comes back later.

When The Baton Drops

Sometimes you can't even reach a login prompt to run those commands. That is what rescue and emergency modes are for, and you reach them by editing the kernel command line from the GRUB menu. At the menu, highlight your normal entry, press e to edit it, find the line that starts with linux, move to its end, add one of the options below, then press Ctrl-X (or F10) to boot that way just this once.

# rescue: single-user mode, root shell, root mounted, almost nothing else running
systemd.unit=rescue.target
 
# emergency: a root shell even earlier, before most filesystems are mounted
systemd.unit=emergency.target
 
# or pick an older, known-good kernel from the "Advanced options for ..." submenu

rescue.target gives you a root shell with the root filesystem mounted and services stopped, which is enough to fix a bad config. emergency.target goes further down, a shell before most mounts happen, for when the disk itself is the problem. Once you have a shell, journalctl -b tells you where the last boot stalled and you repair it from there.

# <device>     <mountpoint>  <type>  <options>                                     <dump> <pass>
UUID=9c1e...   /             ext4    errors=remount-ro                             0      1
UUID=b7f3...   /data         ext4    defaults,nofail,x-systemd.device-timeout=10s  0      2

For a service that can block boot, cap how long systemd waits before giving up on it. Running sudo systemctl edit myagent.service opens a drop-in override, a small file layered on top of the shipped unit so you never touch the vendor's copy. Set the timeout there.

[Service]
# fail fast instead of hanging boot if the agent can't start
TimeoutStartSec=30s

Boot Is Where Quiet Persistence Lives

Everything in the boot chain runs before your monitoring does. An attacker who can write to a unit file, the initramfs, or GRUB's config gets code that runs early, on every boot, often before the tools that would spot it load. That makes boot prime real estate for persistence. Three habits catch most of it. First, know what starts on its own.

Diff that against what you expect to be running. An unfamiliar entry, or a service you never installed showing up in systemd-analyze blame, is both a performance smell and a security one (remember the odd line in systemctl --failed). Second, rate how well your services are boxed in. systemd-analyze security scores each one on how much of the host it could touch if it were compromised.

The score isn't gospel, but it points you at services running with far more privilege than they need, the ones worth hardening with directives like ProtectSystem=strict, PrivateTmp=yes, and NoNewPrivileges=yes. A network-facing daemon sitting at 9.6 UNSAFE is a fine place to start. Third, keep Secure Boot on. That firmware signature check from the very first leg is what stops a tampered bootloader or kernel from loading at all, which closes the earliest link in the chain.

Next time a box is slow to boot, don't stare at blame. Pull the critical chain, find the one unit sitting on it, and check whether it is a wait-online you can disable or a mount you can mark nofail. That single line is usually the whole story, and it is the same line an attacker would love you never to read.

Try this

Work through “Boot Is Where Quiet Persistence Lives” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: one bad line in fstab can hang the whole boot. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related