Centralized logging with journald and rsyslog
Make the journal persistent, forward it to a central host, and keep logs you can actually search after an incident.
On systemd hosts, journald is the default log sink — kernel messages, service stdout, authentication events, and audit dispatcher output land in a structured binary journal. By default on many images the journal is volatile: reboot erases history, and /var/log/journal never gets created. Persistent storage plus forwarding to rsyslog or a log aggregator turns journald from a troubleshooting convenience into an evidence source after compromise.
Immutable infrastructure images should bake Storage=persistent into the golden AMI — do not rely on first-boot Ansible playbooks that might fail silently. Verify with journalctl --disk-usage after every deploy before declaring logging production-ready.
This note enables persistent journals, caps disk usage sanely, forwards JSON to a central host, and shows query patterns with journalctl. For detection pipelines built on those logs, continue with Linux detection engineering and Detection engineering.
Persistent journal locally first — forwarders drop messages if the disk buffer is empty on reboot before persistence is configured.
Enable persistent journal storage
Edit /etc/systemd/journald.conf: set Storage=persistent and Compress=yes. Limit SystemMaxUse= so a runaway debug service cannot fill the root volume. After restart, confirm /var/log/journal/ exists and grows under load.
RateLimitIntervalSec and RateLimitBurst stop broken services from flooding the journal and crowding out auth logs during incidents. Set SyncIntervalSec lower on security-sensitive hosts if you accept slight write amplification for faster flush to disk.
[Journal]Storage=persistentCompress=yesSystemMaxUse=2GMaxRetentionSec=4weekForwardToSyslog=yes
Forward to central rsyslog
On the collector, use imjournal or receive syslog over TLS. Preserve __CURSOR fields where possible for replay detection. Tag forwarded lines with hostname and boot id so you can distinguish duplicate shipper restarts from real duplicate events.
Grafana Alloy and Vector replace classic rsyslog chains on newer deployments — same goal: structured fields, TLS, backoff on collector outage. Keep at least seven days local retention even when forwarding works; collectors fail during the same incidents you need logs for.
# On app servers — forward everything*.* @@logs.example.com:6514# On collector — separate by hostname$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"*.* ?RemoteLogs
journalctl -u ssh -S "2026-07-01" -U "2026-07-02"Jul 01 14:22 sshd[8821]: Accepted publickey for alicejournalctl -b -1 -p errPrevious boot errors — survives reboot if persistentjournalctl _UID=0 --since todayAll root-uid messages todayWhere this goes next
Ship forwarded logs into Loki or Elasticsearch with structured labels. Build detections on SSH and sudo patterns from Linux detection engineering. Pair host logs with Prometheus alerts on disk usage for /var/log/journal so retention caps do not silently delete evidence during an active incident.
journalctl -o json-pretty exports for ticket attachments preserve metadata better than copy-paste from less. Train responders on -u UNIT --since filters before incidents — muscle memory beats reading man pages during a breach.