Logs: the journal and /var/log
journalctl filters, text logs and rotation.
Most of what happens on a server is written down twice: in the systemd journal, and on Ubuntu and RHEL also in plain text files under /var/log. This lesson shows how to get from a symptom to the lines that explain it with journalctl filters, how to read the fields each journal entry carries, who is allowed to read the logs, where the text files are on each distribution, and how old logs are rotated away. Logs are also your record of who did what, so the same skills answer security questions such as "did anyone try to log in overnight?".
One unit and a time window
The systemd lesson used journalctl -u to read one service's messages. Add a time window and you have the question most investigations start with. In the example, someone reported failed SSH logins at about 08:40, so you note the time and ask the SSH service's unit, ssh on Ubuntu (sshd on RHEL), for everything since then. The lab produced these events from addresses in the documentation ranges (a real server would show real addresses): one machine tried three accounts, and an administrator, ess-journald-alice, logged in from another. On your own machine, ssh nosuchuser@localhost (answer the host-key question, then press Ctrl-C at the password prompt) produces an Invalid user line of your own to search for.
--since (and --until) take a time such as 08:39:49, a date and time such as "2026-09-27 06:00", or words such as today, yesterday and "1 hour ago". Each line shows the time, the host, the program that logged it with its PID in brackets, and the message. Since OpenSSH 9.8 the per-connection messages come from sshd-session, not sshd. Invalid user admin means the account does not exist; Connection closed by ... [preauth] means the client gave up before it authenticated; authenticating user root means the account exists and the client disconnected before authenticating (here, because its key was not accepted); Accepted publickey is a successful login, with the fingerprint of the key that was used, and the pam_unix lines mark the session opening and closing.
At a terminal, journalctl opens its output in less; -e jumps to the end, -n 20 shows the last 20 entries, -r puts the newest first, and -f follows new entries as they arrive until you press Ctrl-C. -g keeps only messages that match a regular expression, and a pattern written in lowercase matches regardless of case:
Priorities: finding the errors
Every entry has a priority, a number from 0 to 7: 0 emerg, 1 alert, 2 crit, 3 err, 4 warning, 5 notice, 6 info and 7 debug. Lower means more urgent, and -p err shows err and everything more urgent (0 to 3). When you do not yet know which service is in trouble, start there:
A unit called "Nightly backup" failed to start (the lab created it as a stand-in for a real backup job). The error-level line names the unit but not the reason, so ask the unit for its whole story:
tar could not create its archive because the directory /srv/ess-journald-backup does not exist, and it exited with status 2. Those tar lines did not appear under -p err: systemd logs whatever a service prints at priority info unless the program says otherwise. So use -p err to find the unit, then -u to read why. -b limits any query to the current boot and -b -1 to the one before, which is what you want after a crash and reboot.
The fields behind each line
A journal entry is more than the line journalctl prints. -o verbose shows every field:
Fields whose names start with an underscore are added by journald itself from what the kernel reports about the process that sent the message, so a program cannot fake them (for a process that exits within milliseconds journald can be too late to read them, and they are then missing): _PID, _UID (0 here, because the privileged sshd-session process runs as root), _COMM (the command name), _EXE, _CMDLINE and _SYSTEMD_UNIT. MESSAGE, PRIORITY and SYSLOG_IDENTIFIER come from the program and say whatever it chose. Any field can be a filter: journalctl _COMM=sudo lists every sudo command, and journalctl _UID=1001 everything one user's processes logged. PIDs are reused after a process exits, so to follow one process add its boot, as in journalctl _PID=83656 _BOOT_ID=60492fd2965b48179fa78e0894522e50.
For a timeline, especially one you will compare with other servers, print full dates with the time zone offset: -o short-iso does that, and --utc converts everything to UTC.
Who may read the logs
The journal's files are in /var/log/journal, owned by the group systemd-journal. The + at the end of the mode means the directory also carries an ACL (access control list), extra permission entries for named users or groups on top of the nine bits, and getfacl prints them:
The group:adm:r-x entry lets the adm group in as well, and the default: lines hand the same entries to the files created inside. So members of systemd-journal or adm can read everything, and Ubuntu puts the first administrator account in adm, which is why deploy has read these logs without sudo. On RHEL 10 the same ACL also names wheel (shown below), so every RHEL administrator reads the whole journal without sudo. To see what an ordinary account gets, the lab created ess-journald-bob with sudo useradd -m (any throwaway account you create works the same) and ran commands as him with sudo -u:
ess-journald-bob is in neither group. He may still run journalctl, but it would show only his own processes' messages. It prints a hint, and because he has never logged in and so has no journal file of his own, it reports that it could open no files at all; the text log refuses him outright. The fix is not to loosen file permissions. Use sudo, or give a colleague who reads logs for a living one of the reading groups: sudo usermod -aG systemd-journal NAME for the journal only, or adm for the journal and the text files (the lesson "Reading and editing files safely" showed their permissions). The change takes effect at their next login.
Kept across reboots, and copied to text files
--list-boots lists every boot the journal still has, so this journal survived the machine's reboots, one the evening before and one a few minutes before these commands ran. On Ubuntu 26.04 journald stores its files on disk in /var/log/journal by default: Ubuntu's systemd 259 is built with Storage=persistent as the default. RHEL 10's default is Storage=auto, which writes to disk only if /var/log/journal exists, and RHEL creates it, so the journal is kept there too (this lab machine has booted only once, hence one entry). Rocky's minimal install has no getfacl, so the ACL is read from the systemd rule that sets it:
Where the directory is missing, on some minimal or container images, the journal lives in /run/log/journal in memory and is lost at every reboot. Check with --list-boots before you need a previous boot's logs. journald limits its own size, by default to 10 percent of the filesystem and at most 4 GiB; --disk-usage shows the current total, and the hardening course sets retention deliberately.
Both distributions also run rsyslog, which receives a copy of every message and writes text files. On Ubuntu a drop-in in /usr/lib/systemd/journald.conf.d sets ForwardToSyslog=yes for it, and rsyslog writes /var/log/syslog (almost everything), /var/log/auth.log (logins, sudo, SSH) and /var/log/kern.log, with full timestamps:
RHEL writes /var/log/messages and /var/log/secure instead, readable by root only, and keeps the older timestamp format with no year and no time zone:
Text files are handy for grep and for tools that expect them; the journal has the fields. Neither is safe from someone who has root on the machine, who can delete or edit both, which is why the hardening course forwards logs to another host.
Rotation: how old text logs go away
Text logs would grow forever, so logrotate renames them on a schedule and starts new ones. Its main settings are in /etc/logrotate.conf, and each package adds a file to /etc/logrotate.d. This is rsyslog's:
weekly and rotate 4 keep the current file plus four older ones, so about five weeks of history. After a rotation auth.log starts empty, the previous week becomes auth.log.1, and older copies are compressed to auth.log.2.gz and so on; delaycompress leaves .1 uncompressed for a week because a program may still be finishing writes to it. missingok and notifempty skip missing and empty files, and the postrotate script tells rsyslog to reopen its files. The journal is not rotated by logrotate; journald manages its own files.
A systemd timer runs logrotate once a day (timers are the next lesson), and logrotate -d shows what it would decide without changing anything. That dry run is the quickest answer to "why has this log not been rotated?".
Try this
Write your own entry at warning priority with logger -p warning -t ess-try "disk check: /var is 91% full", then find it with journalctl -t ess-try -p warning (-t filters on the identifier) and in /var/log/syslog with grep ess-try. Look at it with journalctl -t ess-try -o verbose and find PRIORITY=4, SYSLOG_IDENTIFIER=ess-try and your own _UID. Then log a second message at -p info and predict which command shows it: journalctl -t ess-try lists both, -p warning only the first.
Takeaway
Narrow before you read: a unit with -u, a window with --since, and -p err or -g for the lines that matter. Check journalctl --list-boots on every server you look after, so you know the logs you will need are being kept.