systemd units & timers
Start, enable, and inspect services.
Every big apartment building has a superintendent. When the building wakes up in the morning, the super decides what gets switched on and in what order: boiler first, then the elevators, then the lobby lights. If the elevator jams at 2 a.m., the super is the one who resets it. On a Linux machine, that superintendent is systemd.
systemd is the init system, the very first program the kernel (the core part of the operating system that talks directly to the hardware) starts when the machine boots. It runs as PID 1 (process ID 1, the number handed to the first process), and every user-space process on the box traces back to it. Its job is to start background programs, keep them alive, put them in the right order, and restart them when they fall over. Those background programs are called services, and systemd wraps each one in a unit. You give systemd orders through a single command, systemctl (systemd control).
The Verbs You Will Use Every Day
The good news: the same handful of verbs works for every service on the system. Whether you are poking at the web server, the SSH (Secure Shell, the encrypted remote-login service) daemon (the background program that answers those logins), Docker, or an app you wrote yourself, the commands are identical. Learn them once.
start, stop, restart and reload change what is happening right now. status is the one you will run most. Here is the SSH daemon on a web server that has been up a couple of hours.
Read that top to bottom and it answers real questions. The Loaded line shows the path to the unit file and, in parentheses, whether it is enabled (will it start at boot). The Active line shows it is running and for how long. Main PID is the process to look for in a ps listing. The CGroup block (control group, the kernel feature systemd uses to fence off and track every process that belongs to one service) lists everything systemd counts as part of this service, and that matters for security: if something extra is living under a service's cgroup, that is worth a hard look. The last lines are pulled straight from the journal (systemd's central log), so you see why a service behaved the way it did without opening a single log file.
Start Now vs Enable at Boot
This is the distinction that trips up almost everyone. start is the light switch on the wall: it turns the service on this instant. enable is the timer you wire into the porch light so it comes on by itself every evening. They are separate controls. A service can be on now but not wired for the evening, or wired for the evening but currently off.
Two commands answer the two questions separately. is-active asks 'is it running this second?' and is-enabled asks 'will it come back after a reboot?'
That output is a trap waiting to spring. node_exporter is running right now, so a quick glance says everything is fine. But it is disabled, so the next reboot kills it and nothing brings it back. For anything you actually depend on, you want both green, and the one command that does both is systemctl enable --now node_exporter: enable it and start it in a single step. For a defender, the reverse case is the interesting one. A service that is enabled that nobody remembers enabling is exactly how an intruder arranges to survive a reboot.
Reading and Writing a Unit File
A unit file is a recipe card. It says what to cook (which program to run), who cooks it (which user), what to do if it burns (restart or not), and when in the meal it belongs (start order). They live in two places, and the difference matters. Files under /lib/systemd/system come from packages you installed. Files under /etc/systemd/system are yours, and yours win when the names collide. So your own app goes in /etc.
[Unit]Description=My App APIAfter=network.target[Service]User=appuserGroup=appuserExecStart=/usr/local/bin/myapp --port 8080Restart=on-failureRestartSec=5# --- cheap hardening ---NoNewPrivileges=trueProtectSystem=strictProtectHome=truePrivateTmp=trueReadWritePaths=/var/lib/myapp[Install]WantedBy=multi-user.target
Three sections. [Unit] holds the description and ordering: After=network.target tells systemd not to start your app until the network is up. [Service] is the meat. ExecStart is the exact command to run. Restart=on-failure means if the process exits with an error, systemd waits RestartSec seconds and brings it back, so a crash at 3 a.m. heals itself. [Install] is what enable reads: WantedBy=multi-user.target hooks the service into the normal multi-user boot, which is why enabling it makes it start.
daemon-reload is the step people forget. systemd caches unit files in memory; after you write or change one, it keeps using the old copy until you tell it to re-read from disk. The symlink in the output is literally what 'enabled' means under the hood: enabling a service creates a symlink from a boot target's .wants directory to your unit. Disable it and that symlink disappears.
One rule keeps all that hardening from evaporating: never edit a packaged unit in place. Files under /lib/systemd/system (or /usr/lib/systemd/system on newer releases) belong to the package, so the next apt upgrade overwrites them without asking and your changes quietly vanish. Instead run sudo systemctl edit nginx, which opens a small drop-in file at /etc/systemd/system/nginx.service.d/override.conf that layers on top of the vendor unit and survives upgrades. systemctl edit reloads systemd for you when you save. If you ever change a unit file by hand in an editor instead, you have to run systemctl daemon-reload yourself, or systemd keeps using the old definition.
Locking a Service Down
When a contractor shows up to fix one sink, you hand them the key to that one bathroom, not the master key to the whole building. Running a service as root is handing it the master key. If that program is ever tricked into running attacker input, and web-facing programs get tricked constantly, it already owns the machine. The unit file is where you hand out the small key instead, and most of it costs one line each.
User=appuser runs the process as an unprivileged account you made for it. NoNewPrivileges=true stops the process (and anything it spawns) from ever gaining more privileges, so a setuid trick goes nowhere (setuid is a flag on a program file that makes it run with its owner's privileges, often root, instead of yours). ProtectSystem=strict makes the whole filesystem read-only to the service except for the paths you list in ReadWritePaths. ProtectHome=true hides /home, /root and /run/user completely. PrivateTmp=true gives the service its own throwaway /tmp so it cannot watch or tamper with other programs' temp files. None of these need code changes. You add lines and restart.
systemd can grade your work. systemd-analyze security scores a unit from 0 (locked down) to 10 (wide open) and shows which knobs you left loose.
A bare service running as root with no hardening lands around 9.5 and prints UNSAFE. The unit above scores far lower on the strength of five extra lines. This is the cheapest security work you will ever do, and running it on every service you own is a good habit to build. The exposure that remains here is mostly network access, which an app that answers HTTP (the web request protocol) genuinely needs.
Timers Instead of Cron
cron (the classic Unix job scheduler) is a sticky note on the fridge that says 'run backup at 2 a.m.' It runs the job and forgets it happened. A systemd timer is a calendar reminder that also keeps the receipt: the same run lands in the journal, you can ask when it last fired and when it fires next, and a run missed while the machine was off gets caught up. A timer is two files that share a name: a .timer that says when, and a .service that says what.
[Unit]Description=Nightly database backup[Timer]OnCalendar=*-*-* 02:00:00Persistent=trueRandomizedDelaySec=15min[Install]WantedBy=timers.target
OnCalendar uses a plain grammar: *-*-* 02:00:00 means every day at 2 a.m. Persistent=true is the catch-up feature, so if the machine was asleep at 2 a.m. the job runs the moment it wakes. RandomizedDelaySec=15min smears the start time across a window so a fleet of servers does not stampede your backup target at the same second. The matching backup.service is an ordinary unit with Type=oneshot (it runs, finishes, and is not expected to stay alive). You enable the timer, not the service.
Notice the NEXT column reads 02:07, not 02:00: that is the random delay at work. backup.timer shows n/a for its last run because you only just enabled it, so it has not fired yet; the other two have already run today. list-timers is your at-a-glance schedule for the whole box. To see what a run actually did, journalctl -u backup.service (the command that pulls one unit's log lines out of the journal) shows its output with timestamps.
Try this
Work through “Timers Instead of Cron” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: units and timers are a favorite hiding spot. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.