Logs with journald
Find the log you need, fast.
A busy office keeps one sign-in book at the front desk. Every visitor writes the time, their name, and why they came, all in one place, in order. When something goes wrong, security reads that one book instead of hunting for sticky notes stuck on forty different doors. journald (systemd's logging service, where systemd is the program that starts and supervises the services running on a modern Linux machine) is that sign-in book for your whole system.
Every program that starts, stops, crashes, or complains hands a line to journald, and journald writes it into a single store called the journal. You read that store with a command named journalctl. The old approach scattered logs across dozens of plain text files, each with its own format and its own folder. The journal keeps one store instead, and it labels every entry: which service wrote it, the process ID (PID, the number the system gives each running program), the user ID (UID, the number that identifies a user account), the time down to the microsecond, and how urgent the message is. Those labels are the point. Because each line is tagged, you can ask a sharp question like 'show me the web server's errors since 2pm' and get an exact answer, instead of grepping through text files and hoping. One catch: the journal is a binary file, not text, so you cannot open it in an editor. You go through journalctl.
The moves that solve most problems
Four flags handle most of your day. -u picks one service (its systemd 'unit'). --since and --until draw a time window. -p filters by how urgent a line is. -f follows the log live, the way tail -f does. Start with -u, because scoping to one service cuts the noise by ninety percent. Here is every line the Secure Shell server (SSH, the standard way to log into a machine over the network) has written:
That is the calm version. Now the interesting version. Bound it to the last part of the afternoon and pull out only the lines that mention a failed login, using -g (a built-in search that matches the message text with a regular expression):
Three failures in four seconds, all from one address, all guessing common usernames. That shape, many failures from a single source in a short window, is a password-guessing attack (a 'brute force', trying login after login until one sticks). On any machine facing the internet you will see this constantly, and mostly you ignore it. What matters is whether a Failed line is ever followed by an Accepted line from the same address. That is the difference between noise and a break-in.
Priority: a severity dial
Every log line carries a severity tag, the way a hospital sorts patients at the door: a few need attention now, most can wait. The scale has eight steps, from most to least urgent: emerg (system is unusable), alert, crit, err, warning, notice, info, and debug (chatty developer detail). The -p flag sets a floor, not an exact match. Asking for -p err gives you err and everything worse than it (crit, alert, emerg), and hides the softer levels below. This trips people up, so say it slowly: a lower number means more urgent, and -p keeps that end. Here are the error-level lines since the machine booted:
Reading without drowning
A few flags keep long output manageable. -n 50 shows the last fifty lines. -e jumps straight to the end, the newest entries. -r reverses the order so newest sits on top. -f follows live, so you can watch new lines land while you reproduce a bug in another window. -k shows only kernel messages, the same thing the old dmesg command prints. -x pins a short plain-English explanation onto known messages. And -b scopes to the current boot, while -b -1 means the previous boot, which is exactly what you want after a server crashed and rebooted: you read the boot that died, not the fresh one. That last trick only works if the journal survives reboots, which we will come back to.
Every line carries labels
The labels do real work. Ask journalctl to print one line in full, as JSON (a plain text format of key-and-value pairs), and you see everything the journal stapled to it:
Look at which keys start with an underscore, and think back to the sign-in book. A visitor writes their own name on their own line, but the guard at the desk is the one who notes the exact time and which door they walked through, and no visitor can talk the guard out of what the guard saw. The journal splits its fields the same way. The free-text MESSAGE is whatever the program felt like saying. The underscore fields (_PID, _UID, _HOSTNAME, _SYSTEMD_UNIT) are stamped by journald itself, taken from the kernel's own record of which process opened the connection, so the program writing the log cannot forge them. That is why a defender trusts the underscore fields more than the message text. You can filter on any of them: journalctl _SYSTEMD_UNIT=ssh.service, or journalctl _UID=1000 for everything a given user's processes logged, or journalctl _PID=3124 to follow one exact process across every service it touched. That is how you reconstruct a single intruder's session out of a busy log.
The journal is not the whole story
Not everything lands in the journal. Plenty of programs still write their own plain text files under /var/log, and some write to both. The web server nginx writes its access and error logs to /var/log/nginx/. Authentication events get mirrored into a text file too: /var/log/auth.log on Debian and Ubuntu systems, /var/log/secure on Red Hat and Fedora ones. General system messages land in /var/log/syslog. These are ordinary text, so less, tail, and grep all work on them. For a security investigation the auth file is the one you reach for first, because it holds logins, SSH activity, and every use of sudo (short for 'superuser do', the command that runs something with administrator power):
Logs are evidence
On a machine that has been broken into, the logs are your best witness and the attacker's first target at the same time. The failed guesses, the moment a login succeeded, and every sudo command that followed all sit in the journal and in /var/log/auth.log. A careful intruder knows this, so an early move is to erase or edit the record and walk out clean.
The binary journal makes casual tampering harder, since you cannot open it in a text editor and quietly delete a line. It does not make it impossible. Anyone with root can rotate and wipe the journal (journalctl --rotate then journalctl --vacuum-time=1s clears old entries), or delete the folder under /var/log/journal outright. The plain text files under /var/log are even easier to trim. So the rule is blunt: logs stored on a host you already suspect is compromised cannot be fully trusted, because whoever owns the box can rewrite them.
Two defenses follow from that. Ship the important logs off the machine, to a central collector the attacker does not control, so there is a copy they cannot reach (the detection course covers how). And keep logs long enough to matter, because you often find a breach weeks after it happened, and a log you never kept is a question you cannot answer. Both live in one small config file:
[Journal]Storage=persistentSystemMaxUse=2GMaxRetentionSec=1monthForwardToSyslog=yesSeal=yes
Storage=persistent forces the journal onto disk under /var/log/journal, so it survives reboots instead of living only in memory. SystemMaxUse caps how much disk it may use. MaxRetentionSec keeps a month of history. ForwardToSyslog hands a copy to rsyslog (an older, separate logging service) so those plain text files stay populated. Seal=yes turns on Forward Secure Sealing (FSS), which works like a wax seal on an envelope: it does not stop someone opening the letter, but it shows plainly that they did. FSS cryptographically seals the journal at set intervals, so any entry edited after its seal shows up as tampered. You set the keys once with journalctl --setup-keys and check integrity later with journalctl --verify. FSS does not stop someone deleting the journal, but it makes silent edits show up. To see how much space the journal is using right now:
The first thing to run on an unfamiliar server that is misbehaving is journalctl -p err -b -e. It drops you at the newest error-level lines of the current boot, and most of the time the failing line is already on the screen, waiting for you to read it.
Try this
Work through “Logs are evidence” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: a reboot can wipe your evidence. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.