Logs and clocks you can trust
journald, rsyslog, forwarding and chrony.
Logs are the record you rely on after something goes wrong: who logged in, what ran as root, which service changed and when. That record is only worth something if it survives a reboot, cannot fill the disk, is readable only by the people who need it, also exists somewhere an intruder on the server cannot reach, and carries timestamps you can line up with other machines. In this lesson you check each of those on a default Ubuntu server, set a size limit for the journal, forward logs to a log host over TLS, and read chrony's view of the clock. Reading the journal and /var/log is covered in Linux essentials; here the question is whether you can trust them.
What the defaults already give you
"Logs: the journal and /var/log" in Linux essentials showed where a default server keeps its logs: a persistent journal, and rsyslog's text copies (auth.log and syslog on Ubuntu, secure and messages on RHEL), readable by the adm and systemd-journal groups. Three details of that layout matter for hardening. The first is how large the journal may grow before you set anything:
"max 2.2G" is journald's computed default: 10% of the filesystem, capped at 4 GiB, while journald also keeps 15% of the filesystem free. The second is who can read the logs:
Every member of these groups reads the whole journal without sudo, and adm members read auth.log and syslog too. Here systemd-journal is empty and adm holds syslog (rsyslog's own account) and deploy. That makes membership a privilege worth reviewing like sudo: the logs hold user names, source addresses and every command line run through sudo, including a password someone typed as an argument by mistake. Ubuntu puts the first administrator in adm, which is why deploy is listed. The third is how rsyslog gets its copy. Ubuntu forwards every message from journald to rsyslog; on RHEL 10 rsyslog reads the journal itself, with the imjournal module:
Either way rsyslog receives the messages journald collects, from services, the kernel and logger alike, so forwarding from rsyslog, as configured below, sends the whole record and not only what programs write to syslog directly.
A size limit you choose
The threat runs in two directions. A flood of messages can fill /var and take services down with it, and a journal that rotates too quickly deletes the evidence you need next week. journald's defaults already cap the journal, and its rate limit (10,000 messages per 30 seconds per service by default) drops messages only from the service that floods. What the defaults cannot know is how much history you need locally. Set it explicitly with a drop-in; as everywhere in this course, comments go on their own lines.
[Journal]# SecOpsLog: at most 1 GiB of journal on /var/log/journal, and never# leave less than 2 GiB free on that filesystem. The smaller limit wins.SystemMaxUse=1GSystemKeepFree=2G
Restarting journald is routine; systemd holds its sockets open during the restart. The new line proves the drop-in was read: "max 1G", and "882.6M free" is the room left under that limit, not free disk space. The impact is that journald deletes the oldest archived files once the journal reaches 1 GiB, so size it for the history you want to search on the machine itself and rely on the off-host copy below for the rest. To roll back, delete the file and restart systemd-journald. The explicit limit is SecOpsLog advice; persistent and compressed storage, which the CIS RHEL 10 Level 1 server profile (scap-security-guide 0.1.82) requires, are already the defaults.
Copy the logs off the host
An intruder who gains root can edit or delete every log on the server. The only record they cannot touch is one that already left it. rsyslog's omfwd action sends each message to a log host as it is written, and TLS protects it on the way: plain syslog over UDP or TCP can be read and forged by anyone on the path. TLS support is a separate package, rsyslog-gnutls (on both Ubuntu and RHEL; RHEL also offers rsyslog-openssl). A second machine running rsyslog can be the log host. The lab builds one on the same VM: a network namespace at 203.0.113.2, wired to the server (203.0.113.1) with a virtual cable (a veth pair), stands in for it.
The log host needs a certificate. In production it comes from your internal CA; the lab makes a CA and a certificate for loghost.example.test, valid for seven days:
The receiver's configuration accepts TLS on port 6514 and writes each sending host's messages to its own file; anon means it does not ask the senders for certificates. Write it with sudoedit, create its two directories (sudo install -d -m 0700 /var/spool/rsyslog/hard-log-receiver and sudo install -d -m 0750 -o root -g adm /var/log/hard-log-remote), and install rsyslog-gnutls. It runs as a second rsyslogd inside the namespace, started as a transient systemd unit:
Now the sending side, on the server:
# SecOpsLog: send a copy of every message to the central log host,# encrypted with TLS on port 6514. The log host must present a# certificate for loghost.example.test signed by the CA in ca.pem.# While it is unreachable, messages wait in a queue that spills to# disk (/var/spool/rsyslog, at most 1 GB) and rsyslog keeps retrying.action(type="omfwd" target="203.0.113.2" port="6514" protocol="tcp"StreamDriver="gtls" StreamDriverMode="1"StreamDriverAuthMode="x509/name"StreamDriverPermittedPeers="loghost.example.test"StreamDriver.CAFile="/etc/rsyslog.d/secopslog-tls/ca.pem"queue.type="LinkedList" queue.filename="fwd_loghost"queue.maxDiskSpace="1g" queue.saveOnShutdown="on"action.resumeRetryCount="-1")
StreamDriverMode="1" means TLS only. x509/name with the permitted peer makes rsyslog check that the certificate names the log host, so a machine that merely answers on 203.0.113.2 is refused. In production target would be the log host's DNS name. The CA file sits under /etc/rsyslog.d because Ubuntu's AppArmor profile for rsyslogd lets it read configuration from there and not from arbitrary paths. The queue settings decide what happens during an outage: LinkedList with a queue.filename is an in-memory queue that can spill to disk, and a retry count of -1 means retry forever instead of discarding.
rsyslogd -N1 checks the whole configuration without starting anything; run it before every restart. ss shows the established connection to port 6514, and the log host has the test message under this server's name, web01. On a real log host you would read its files there; the lab's are visible on the same VM because the namespace shares the filesystem. Now stop the log host, log during the outage, and start it again:
Both messages arrived once the log host was back: rsyslog noticed the broken connection, kept the messages in the action's queue and retried until it could deliver them. TCP forwarding can still lose a message sent into a connection that has just died without either side noticing; where every message must arrive, rsyslog offers RELP (package rsyslog-relp), which acknowledges each message.
This setup authenticates only the log host. The log host accepts any client that reaches port 6514, so root on any machine can write forged lines under any host name or fill its disk. For evidence you can rely on, use mutual TLS: give each server a client certificate, set the receiver's StreamDriver.AuthMode to x509/name with the permitted peers listed (rsyslog's imtcp documentation), rate-limit per source, and remember that the log host knows the authenticated peer, not the host name written inside the message.
The operational impact is a network dependency (allow TCP 6514 to the log host) and disk use in /var/spool/rsyslog while the log host is down. To roll back, delete the file and restart rsyslog. The alternative is systemd-journal-upload, which sends the journal over HTTPS to systemd-journal-remote. The CIS RHEL 10 Level 1 server profile takes that route: journald persistent and compressed, ForwardToSyslog=no, journal upload enabled, and rsyslog not running alongside. On Ubuntu, ForwardToSyslog=no would stop feeding rsyslog and empty auth.log. SecOpsLog advice: forward with whichever protocol your log platform receives, and send each message once.
Time you can trust
A log line is only as useful as its timestamp. You correlate events across machines by time, TOTP codes (the SSH MFA lesson) stop working when the clock drifts, TLS rejects certificates that seem not yet valid, and Kerberos by default refuses clocks that disagree by more than five minutes. Both platforms keep time with chrony; systemd-timesyncd is not installed on Ubuntu 26.04.
"System clock synchronized: yes" with "NTP service: active" is the healthy state: chrony runs and has told the kernel that the clock is right. chronyc tracking names the reference (185.125.190.122, one of Canonical's time servers), the stratum (3, one step below that stratum 2 server), how far the system clock is from NTP time (about 2 ms here), and "Leap status : Normal". A clock that cannot synchronise is a finding in itself: alert on "Leap status : Not synchronised". chronyc sources -v prints a legend with the details.
^ means a server; * is the current best source, + a source combined with it, - a usable source that is not combined, and ? would mean unusable. Reach is an octal register of the last eight polls: 377 means all eight answered, 17 would be only the last four (as right after a restart, below), and 0 none. The five addresses are Canonical's time servers, and here is how Ubuntu 26.04 talks to them:
Ubuntu 26.04 ships Network Time Security (NTS) by default. NTS authenticates time replies, so a machine in the network path cannot shift your clock. Ubuntu's source file also marks those servers prefer. Before chrony sends NTP on UDP 123 it performs an NTS key exchange (NTS-KE) over TCP 4460 to get cookies; authdata shows it done for every source: mode NTS, no failed attempts (Atmp 0, NAK 0) and Cook 8, eight cookies in hand. The two journal lines from earlier in this boot are the failure mode: the key exchange with two servers timed out, and chrony retried until it succeeded. Where a firewall blocks outbound TCP 4460, those lines keep coming, Cook stays 0 and every source shows ^? with Reach 0, even with UDP 123 open. Allow both.
Many networks run an internal time server. The lab runs one in the namespace: a second chronyd that serves the VM's own clock at stratum 8 and offers NTS, with a certificate for 203.0.113.2 from the lab CA. First add it as a plain NTP server. Files in /etc/chrony/sources.d must end in .sources and hold only server, pool and peer lines; chronyc reload sources applies them without a restart.
# SecOpsLog lab: the time server in the lab's network namespace.server 203.0.113.2 iburst
The server answers (Reach is non-zero), yet chrony does not use it: - means not combined, and selectdata shows why. Its state P means another selectable source is preferred through the prefer option, and the NTS sources' effective options (EOpts) add T and R, trust and require. chrony.conf(5) explains those two: in the default authselectmode mix, when authenticated and unauthenticated sources are both configured, the authenticated ones get require and trust, so an unauthenticated server is used only when it agrees with them. That protects against a forged time source, and it has a consequence: when the NTS sources are unreachable, the requirement cannot be met, the plain server waits (state W in chronyc(1)) and the clock stays unsynchronised. An internal NTP server alone does not rescue an Ubuntu 26.04 machine whose NTS traffic is blocked; open TCP 4460 and UDP 123 to the NTS servers, or give your internal server NTS. The lab server offers it, so switch the source to NTS and tell chrony to trust the lab CA:
# SecOpsLog lab: the time server in the lab's network namespace, with NTS.server 203.0.113.2 iburst nts
# SecOpsLog lab: trust the lab CA for NTS servers' certificates.ntstrustedcerts /etc/chrony/hard-log-ca.pem
Files in conf.d are read only at start-up, hence the restart. authdata now shows a completed key exchange with the lab server too (Cook 8), and selectdata shows it authenticated (Auth Y). With only authenticated sources configured, T and R are gone and the lab server is selectable, still in state P behind Ubuntu's preferred servers; where those cannot be reached, it is the one chrony can use. In production the server line would name the time server's DNS name, its certificate would come from your internal CA, and you would mark it prefer or remove the public pools. To roll back, delete both files and restart chrony.
makestep 1 3 lets chrony step (jump) the clock when it is more than one second off during its first three updates, which fixes a badly wrong clock at boot. After that it only slews, speeding up or slowing down the clock until it agrees, so timestamps never jump backwards on a running server. rtcsync copies system time to the hardware clock. Both are the distributions' defaults. The CIS RHEL 10 Level 1 profile requires a remote time server in chrony's configuration and chronyd running as the chrony user; NTS wherever your time servers offer it is SecOpsLog advice.
Rocky Linux ships a public pool without NTS, and here chrony synchronised to pool servers over plain NTP (^*). To use NTS on RHEL, replace the pool line with servers that offer it, each written as server NAME iburst nts. To undo any source change, remove the line or file and run sudo chronyc reload sources.
Try this
On your Ubuntu lab machine, build the log host as above and add the forwarding file with the CA copied to /etc/rsyslog.d/secopslog-tls/ca.pem. Stop the receiver, run logger -p auth.warning "during the outage" twice, start the receiver again, and confirm that both messages arrive in its file. Then run chronyc sources -v and sudo chronyc -N authdata and explain every state symbol you see, and what Cook tells you about NTS.
Takeaway
A log you can trust is persistent, sized on purpose, readable by few, copied off the host over TLS as it is written, and stamped by a clock that chrony reports as synchronised. Check the last two from the receiving end and from chronyc tracking, not from the configuration file.