Units, dependencies & targets
How systemd orders and wires the system.
A Linux server booting up works like a restaurant opening for the day. Before the first customer sits down, the power has to be on, the gas lit, the walk-in fridge cold, the till counted. Some of those steps have a strict order: you cannot light the gas range before the gas line is open. Others can happen whenever, and the sequence does not matter, like folding napkins. Somebody has to hold that whole checklist in their head and run it in a sane order.
On a modern Linux machine, that somebody is systemd, the program that starts, stops, and supervises everything else. It runs as process ID 1 (PID 1 for short), the very first process the kernel hands control to once it finishes loading. The kernel, the core of the operating system that talks straight to the hardware, starts exactly one program, and that program is systemd. systemd does not keep a flat to-do list. It models the whole machine as a graph, a web of small managed things called units, each one declaring how it relates to the others. Learn to read that graph and you can explain any boot order, any failed service, and any 'why is this thing even running' question a security review throws at you.
Everything Is a Unit
A unit is one card on the manager's board, a single thing systemd knows how to start, stop, and watch. The file extension on the card's name tells you what kind of thing it is. A .service is a running process, like a web server or a database (these long-running background processes are called daemons, programs with no window that keep running after you log out). A .socket is a listening network port or pipe that can start its service the moment someone connects, a trick called socket activation. A .timer is a schedule, the systemd replacement for cron (the classic time-based job scheduler on Unix). A .mount describes a filesystem being attached. A .target is different: it starts nothing on its own. It is a label for a group of units, a checkpoint the boot passes through, and we get to those below.
Every unit is defined by a plain text file, and where that file lives decides who is allowed to change it, which matters a lot for security. Files under /lib/systemd/system (also seen as /usr/lib/systemd/system on merged-usr systems) ship with your packages. Files under /etc/systemd/system belong to the local administrator, and they win over the package versions. Files under /run/systemd/system are runtime-only and vanish on reboot. To see the real definition of any unit along with the exact path it came from, run systemctl cat, which prints the file path as a comment on the first line. That path is your first clue about whether a unit belongs there.
Two Wires: Ordering and Requirement
Here is the thing people mash together and then lose an afternoon debugging. Two completely separate questions live inside a unit file. One is about sequence: in what order do things start. The other is about presence: does starting me drag another unit in at all. A recipe has the same two axes. 'Add the eggs after the flour' is sequence. 'This cake needs eggs' is a requirement. Neither one implies the other.
Sequence is spelled After= and Before=. These control order and nothing else. After=postgresql.service means 'if postgresql is starting in this same transaction (the batch of units systemd is bringing up together right now), wait for it before you start me.' It says nothing about whether postgresql starts at all. Presence is spelled Wants= and Requires=. Wants= is the soft pull: bring the other unit in, but if it fails, carry on anyway. Requires= is the strict pull: bring it in, and if it fails to start, fail me too. For finer control there is Requisite= (the other unit must already be running, do not start it), BindsTo= (stop me the instant it stops), and PartOf= (send my stop and restart commands down to it as well). Day to day you reach for After= paired with Wants= or Requires=.
Notice that nginx both wants network-online.target (pull it in) and is ordered after it (start once it is up). That pairing is the pattern you copy. Here is a small unit that spells out a real dependency on a database, with a restart policy so a crash does not leave you down.
[Unit]Description=My AppWants=postgresql.service network-online.targetAfter=postgresql.service network-online.target[Service]ExecStart=/usr/local/bin/myappRestart=on-failure[Install]WantedBy=multi-user.target
Targets Are Milestones, Not Runlevels
A target is a checkpoint flag partway around a race track. Reaching it means everything wanted by it is up and running. It starts no process of its own; it is a label, and a bundle of units hang off it. Targets replaced the old numbered SysV runlevels (the fixed 0-through-6 boot states from init, the startup system that came before systemd). The ones you meet daily: multi-user.target is a normal headless server, graphical.target is that plus a desktop and login screen (a full graphical user interface, or GUI), rescue.target is single-user recovery with almost nothing running, and emergency.target is barer still, meant for fixing a root filesystem that will not mount. Boot walks a chain of them: sysinit.target, then basic.target, then multi-user.target, then optionally graphical.target, each pulling in the one before it.
get-default tells you the checkpoint the machine boots to. list-dependencies unrolls the whole tree beneath a target, which is how you audit exactly what a 'normal boot' drags in. To move to a target right now, run systemctl isolate rescue.target, which brings the system to that state live. Be careful with isolate over a remote connection: it stops every unit not wanted by the new target, so isolating rescue.target on a box you only reach by SSH (Secure Shell, the encrypted remote-login service) will drop ssh.service and lock you out. Run it from the physical console, or not at all.
Reading the Boot Order
When boot feels slow, or when you want to know what actually gated startup, systemd hands you the answer. systemd-analyze critical-chain follows the single longest path of ordered units, the chain that decided when the system was finally 'up'. The @ number is the clock time each unit became active; the + number is how long that one unit took on its own. Read down the chain to the biggest + and you have found the thing everyone else was waiting on. For a flat ranking of slow starters, systemd-analyze blame lists every unit by start time. From a security angle, a boot that suddenly got 20 seconds slower, or a brand-new unit sitting on the critical path, is worth a second look.
That chain has a smoking gun: systemd-networkd-wait-online.service took 5.5 seconds, and everything after it had to wait. That service sits and blocks until the network is fully configured, often stuck on a slow DHCP lease (DHCP, the Dynamic Host Configuration Protocol, is how a machine automatically asks the network for an IP (Internet Protocol) address). Now you know why nginx did not answer for the first nine seconds, and you know the fix lives upstream in the network, not in nginx.
Where Attackers Hide, and How You See Them
An attacker who lands on a box wants to survive reboots, and systemd is a comfortable place to hide, because a running machine is already stuffed with units nobody ever reads. A .timer paired with a .service is the systemd version of a cron job: the timer says when, the service says what. Drop those two files into /etc/systemd/system, enable them, and you have a payload that fires on a schedule and comes back after every reboot. Your job as the defender is to know what normal looks like so the abnormal jumps out. Start with the timer list.
The name apt-compat.timer is picked to blend in next to apt-daily, but the cadence gives it away: it fires every ten minutes, while real apt maintenance runs about once a day. Anything named after a package manager but beating on a short, regular interval like a beacon deserves a read. So read it.
Every tell is on screen. OnCalendar=*:0/10 is every ten minutes, not daily, whatever the description claims. The service is a oneshot (a unit that runs one command and exits) that shells out to curl (a command-line tool that downloads whatever a web address points to) and pipes a raw IP address straight into bash, with no domain name anywhere in sight. Genuine distribution units do not download and run code from a bare IP on a timer. To hunt these across a fleet, list everything set to start at boot with systemctl list-unit-files --state=enabled, and run systemd-delta to surface drop-in overrides. A drop-in is a small .conf file under a unit's .d directory that patches a shipped unit without replacing it, like a sticky note stuck onto a printed recipe. That is how an attacker quietly rewrites the ExecStart of a trusted service like ssh.service, by dropping /etc/systemd/system/ssh.service.d/override.conf. The main unit file looks untouched; the behavior is not.
Check Your Work
systemd does not watch your files. Think of the running manager as a chef cooking from the recipe memorized this morning: edit the unit on disk and systemd keeps using the old copy in memory until you tell it to re-read the disk with systemctl daemon-reload. Skip that step and your change silently does nothing, which sends people down a long wrong path. Before you enable anything, run systemd-analyze verify against the file. It parses the unit the way systemd will at load time and surfaces mistakes that systemd would otherwise swallow in silence, like a mistyped directive.
Say you had fat-fingered Requires= as Reqiures= while editing that unit. That one line is the whole point. To systemd a misspelled Reqiures= is not an error, it is a key it does not recognize, so it drops the line and starts your service with no dependency at all. Without verify you would only find out when the app crashed at boot because its database was not up yet. After a clean reload, confirm the manager sees what you meant with systemctl show myapp.service -p After -p Wants, so you are checking the configuration that is actually loaded, not the file you hope it loaded.
Try this
Work through “Check Your Work” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: after= and Requires= are different wires, and each without the other bites. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.